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7th Annual CMMI Technology Conference & User Group 


“Investigation, Measures and Lessons Learned about the Relationship between CMMI® Process Capability and 
Project or Program Performance” 
Denver, Colorado 


November 12 - 15, 2007 


MONDAY, NOVEMBER 12, 2007 


e CMMI V1.2 -- An Overview Mr. David Phillips, SEI 


TUESDAY, NOVEMBER 13, 2007 
State of CMMI® 
e Mr. Clyde Chittister, Chief Operating Officer, SEI 
Executive Panel 
Panelists: 
Ms. Kristen Baldwin, Office of the Secretary of Defense 
Mr. Tom Neff, Defense Threat Reduction Agency 
Mr. Rich Frost, General Motors 
Lunch with Guest Speaker 
e Mr. Mark Schaffer, Director, Systems & Software Engineering, OSD (AT&L) 
Technical Sessions 
TRACK 1 
e When the Only Tool You Have is a Hammer, Every Problem Begins to Look Like a Nail, Mr. Sam Fogle, ACE Guides, LLC 
e The Journey to CMMI Level , Mr. Andrew Lay, Lockheed Martin Aeronautics Company 
e Visualizing Improvement with Capability Waypoints, Mr. Robert Jacob, 
Naval Air Systems CommandiInstitutionalization Measures: Key to Improved Process Monitoring, Dr. John Rusnak, Lockheed Martin Space Systems 
Company 
TRACK 3 
e Assuring Quality for Efficient & Sufficient Testing Mr. Pramod Varma, Wipro Technologies 


TRACK 4 


e Bridging Process Improvement During Program Management Evolution: An Experience Report Capt DeWitt Latimer, USAF 
e An “Embedded SCAMPI-C” Appraisal at the National Security Agency. Mr. Joseph Wickless, SEI 


TRACK 5 


e Linking Project Performance to CMMI Process Capability through Lean Measurements, Mr. Jeffrey Dutton, Jacobs Technology 
e Quantitative Models for Predicting Project Success, Dr. Rick Hefner, Northrop Grumman Corporation 


TRACK 6 
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How to Kick Start a Process Improvement Effort to Achieve a CMMI Rating, Ms. Brenda Hall, Computer Sciences Corporation 

SEI Appraisal Program Quality Report, Mr. William Hayes, SEI 

The Process In-execution Review (PIER) After Three Years, Mr. Dale Swanson, The MITRE Corporation 

I’m Preparing My Organization for an Appraisal, but I’m Not Really Sure I Understand this PIID Thing. Should I Worry?, Mr. Sam Fogle, ACE Guides, LLC 


TRACK 7 


e Aligning CMMI and ITIL — Where Am I and Which Way Should I Go, Mr. Pat Mitryk, Cognence, Inc 
e Integrated System Framework: A Way Out of the Multi-Model Madness, Mr. Paul Bymes, Integrated System Diagnostics 


WEDNESDAY, NOVEMBER 14, 2007 


Lunch with Guest Speaker 

e Ms. Mary Poppendieck, President, Poppendieck, LLC 
Technical Sessions 
TRACK 1 


e CMMI Contenders, CMMI Pretenders, Dr. Rick Hefner, Northrop Grumman Corporation 

e Initial Fears of CMMI Introduction and How Things Really Played Out, Dr. Paul Nugent, General Dynamics Advanced 

Information Systems 

Software Firm + CMMI Level 2 Initiative + 15 months = Dramatic Quality Improvements, Mr. Jeff Simpson, Campus Management Corporation 
How to Explain the Value of Every CMMI Practice, Dr. Rick Hefner, Northrop Grumman Corporation 

Mrs. Doubtfire Answers Your Questions about Process Improvement, Dr. Rick Hefner, Northrop Grumman Corporation 

Developing a Second Generation Directive System Architecture, Mr. Kenneth Weinberg, Raytheon Company 

Whose Processes Are These, Anyway, Ms. Judith Tejan, AAI Services Corporation 

Scientific Breakthroughs in Process Improvement, Ms. Cheryl White, Change Delivery Group 


TRACK 2 


e The What, When, Why and How for CMMI Training, Mr. Tom Bragg, AVISTA Incorporated 

e Transitioning to the CMMI: What They Never Told You, Mr. Steve Fried, The Boeing Company 

e¢ CMMI Implementation: Overcoming the PPQA Challenge, Mr. Pat Mitryk, Cognence, Inc. 

e How to Measurably Improve Your Requirements, Mr. Timothy Olson, Lean Solutions Institute, Inc. (QIC) 


TRACK 3 


Using Lean Six Sigma to Implement CMMI High Maturity Practices, Ms. Beth Clark, Lockheed Martin 

The Potential for Lean Acquisition of Software Intensive Systems, Mr. Jeffrey Dutton, Jacobs Technology 

Lean, CMMI and Six Sigma Working Together to Achieve High Success, Ms. Susan Bassham, US Army Aviation and Missle Command 
Comparing and Contrasting the PP & PMC Process Areas of CMMI v 1.2 and SCRUM, Dr. Aldo Dagnino ABB, Inc. - US Corporate Research 
Effective Systems Engineering: What’s the Payoff for Program Performance?, NDIA Systems EngineeringsEffectiveness 

What’s All this ‘churn’ in Systems Engineering Standards and Models!?, Mr. Donald Gantzer, SAIC 


TRACK 4 


Driving Process Improvement Using the CMMI-ACQ at General Motors, Dr. Richard Frost, General Motors 

Leading Indicators for Acquisition Programs, Mr. Robert Ferguson, SEI 

CMMI High Maturity Misconceptions, Mr. William Hayes, SEI 

High Maturity: How Do We Know?, Dr. Mike Konrad, SEI 

High Maturity System/Software Cost Estimation, Dr. Richard Welch, Northrop Grumman Corporation 

ADVANCE - Implementing a Defect Model for Performance Prediction, Mr. Stanley Martin, L-3 Communications/IS 
Statistically Managing a Critical Logistics Schedule Using CMMI, Mr. Robert Tuthill, Northrop Grumman Corporation 
e A More Practical Set of High Maturity Practices, Dr. Rick Hefner, Northrop Grumman Corporation 


TRACK 5 


e Program Level Return on Investment for CMMI® Process Improvement, Mr. J Perry, BAE Systems 

e How Do We Get on the Road to Maturity?, Mrs. Debra Perry, Harris Corporation 

e Understanding CMMI Measurement Capabilities Performance & Outcomes: Results from the 2007 SEI State of Measurement Practices Survey, Dr. Dennis 
Goldenson, SEI 

e Using Predicted Delivered Defects as a Management Tool, Mr. Dustin Sims, BAE Systems 

e Calibrating the Project Planning Process, Mr. Donald Corpron, Northrop Grumman Corporation 

e All Others Bring Data, Ms. Charlene Gross, SEI 


TRACK 6 


e Executing a Successful CMMI Maturity Level 3 Scampi for Spawar Systems Center Charleston, Mr. Michael Kutch, 
SPAWAR Systems Center Charleston 
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e CMMI SCAMPI Appraisals — The People/The Process/The Results-United Space Alliance, LLC Lessons Learned, Ms. Robin Hurst, United Space Alliance, 
LLC 

e Proposed Approach to Heterogeneous CMMI Appraisals, Mr. Joseph Vandeville, Northrop Grumman Corporation 

e Selecting a Representative Sample for CMMI Enterprise Appraisals, Ms. Kathryn Kirby, Raytheon Company 

e Logistics and Lessons Learned in Conducting an CMMI® Maturity Level 3 Full-Model Scope Enteprise-level Appraisal 
Ms. Kathryn Kirby, Raytheon Company 


TRACK 7 


e Excellence at the Organization, Team and Individual levels; CMMI, TSP and PSP - Experience, Lessons Learned and Why all Three are Needed, Mr. Girish 
Seshagiri, Advanced Information Services, Inc. 

IEEE Life Cycle Standards and the CMMI® — Implementation Considerations, Dr. Peter Hantos, The Aerospace Corporation 

Using CMMI and OPM3 to Improve Performance, Mr. Thomas Keuten, Pariveda Solutions 

Complementary or Competing? Achieving Synergy with OPM3®, CMMI®, and ISO 9001-2000, Mr. Mark Scott, Harris Corporation 

Formal Process Definition with Industry Standards, Mr. Chris Armstrong, Armstrong Process Group, Inc. 

Project Management Architecture Design as a Critical Success Factor in CMMI Model Implementation, Mr. Christen MacMillan, L-3 Communications 


THURSDAY, NOVEMBER 15, 2007 


Lunch and Award Presentation 
TRACK 1 


e Fast Track to Higher CMMI Maturity Levels: Lessons Learned from Five Initiatives, Ms. Cheryl White, Change Delivery 
e Seven Success Factors for CMMI Based Process Improvement, Mr. Orhan Kalayci, XPI - eXtreme Process Improvement 
e« CMMI Process Improvement: It’s Not a Technical Problem, It’s a People Problem!, Mr. Rolf Reitzig, Cognence 

e Improving Project Proposal Quality via CMMI, Mr. Chen Wang, Institute for Information Industry 


TRACK 2 


e A Framework to Manage and Evaluate Remote Software Testing Using CMMI, Dr. Aldo Dagnino, ABB, Inc. - US Corporate Research 
e CMMI, Configuration Management, and Baseball — How to Score, Ms. Julie Schmarje, Raytheon Company 
e Automated Systems for Project Portfolio Management - Project Success and Outstanding Earned Value, Mr. Pothiraj Selvaraj, 

Global Computer Enterprises 


TRACK 3 


e Project Management by Functional Capability, Mr. Fred Schenker, SEI 

e Software Architecture Development Leveraging the Attribute Driven Design and CMMI Methodologies, Dr. Aldo Dagnino, 
ABB, Inc. US Corporate Research 

e Systems Assurance — Practices Make Perfect - How Your Engineering and Management Practices Can Help Meet the Assurance Challenge, Mr. Paul Croll, 
Computer Sciences Corporation 

e Tools and Resources to Enable Systems Engineering Improvement, Mr. Michael Kutch, SPAWAR Systems Center Charleston 

e Applying CMMI Principles to Certification Process of Legacy Aircraft, Ms. Michele Bruno, The Boeing Company 

e Accreditation of Undergraduate Programs in Computing, Software Engineering, Systems Engineering and the Ties to CMMI-based Improvement, Mr. Dan 
Nash, Raytheon Company 

e How Future Trends in Systems and Software Engineering Bode Well for Enabling the Rapid Adoption of CMMI, Dr. Ken Nidiffer, SEI 


TRACK 4 


Thought Before Action: A High Maturity Roadmap for the Lower Maturity Organization, Mr. James McHale, SEI 

Integrated Implementation of Advanced Maturity Practices, Mr. Dale Childs, DFAS 

Process Performance Baselines and Models: Duh, I Don’t Get It, Ms. Diane Mizukami-Williams, Northrop Grumman Mission Systems 

Expanding Statistical Process Control Across All Engineering Disciplines: A Sequence of Practical Case Studies, Dr. Richard Welch, Northrop Grumman 
Corporation 

e Statistical Process Control Applied to Specification Requirements Process, Mr. Al Florence, The MITRE Corporation 

e Implementing High Maturity in a Production Support Environment, Ms. Virginia Slavin, SSCI 

e Using the Scientific Method at Levels 4-5, Dr. Jeff Ricketts, Raytheon Company 


TRACK 5 


The Productivity Puzzle, Mrs. Jill Brooks, Raytheon Company 

Using Metrics to Develop a Software Project Strategy, Mr. Donald Beckett, Quantitative Software Management 

Lessons Leamed in the Implementation of Measurement Techniques for CMMI GP 2.8, Dr. Susanna Schwab, L-3 Communications 
Optimizing the Measurement Process, Mr. Gary Natwick, Harris Corporation 

Measurement Strategies in the CMMI, Dr. Rick Hefner, Northrop Grumman Corporation 

5 Major Sites, 4 Separate Disciplines, 11,500 Engineers, 1 Data Repository: Having Data You Can Actually Use — Priceless! 

Mrs. Jill Brooks, Raytheon Company 


TRACK 6 
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Cutting Appraisal Costs in Half, Dr. Rick Hefner, Northrop Grumman Corporation 

Experiences Implementing Very Large High Confidence Enterprise Appraisals, Mr. Paul Byrnes, Integrated System Diagnostics 
Process Compliance the Smart Way, Mr. Gary Natwick, Harris Corporation 

Judging the Suitability of Alternative Practices, Dr. Rick Hefner, Northrop Grumman Corporation 

Lessons Learned Conducting High Maturity SCAMPIs, Mr. Paul Byrnes, Integrated System Diagnostics 

Benefits of SCAMPI Class C in Small Settings, Dr. Mary Anne Herndon, Transdyne Corporation 

Lower Cost, More Effective Alternatives to SCAMPIs, Dr. Rick Hefner, Northrop Grumman Corporation 

Using Workshops to Speed CMMI Adoption and Evidence Gathering, Dr. Rick Hefner, Northrop Grumman Corporation 


TRACK 7 


Quality Maturity Model — Foundation for Process Institutionalization, Mr. Sumit Gupta, Royal Bank of Scotland - India Development Center 
Not Just for Software Anymore: Lessons Learned From a CMMI™ Appraisal on Projects in a Nonnuclear Weapons Facility, 

Mr. Daniel Fritts, Honeywell 

CMMI for Services Overview, Mr. Craig Hollenbach, Northrop Grumman Corporation 

Defining Lean Service and Maintenance Processes that are CMMI Compliant, Mr. Timothy Olson, Lean Solutions Institute, Inc. (QIC) 
Implementing Acquisition and System Engineering Processes in a Maintenance Organization, Mr. Bill Fetech, The MITRE Corporation 
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ON FERENCE 


D USER GROUP 


“Investigation, Measures and Lessons Learned about the 
Relationship between CMMI® Process Capability and 
Project or Program Performance” 


Sponsored by: 
National Defense Industrial Association, 
Systems Engineering Division 
in conjunction with 
Software Engineering Institute, 
Carnegie Mellon University 


Ae a _ | = Software Engineering Institute 


SATIONS |. DEFENSE ISDLATRIAL ASSCKCIAT ION 


EUSP EEA =— rientation 


STRENGTH THROUGH INDUSTRY & TECHNOLOGY 


Event #8110 
November 12-15, 2007 
Hyatt Regency Tech Center ¢ Denver, CO 


CMMI is registered in the US Patent & Trademark Office by Carnegie Mellon University 


Conference Agenda 


SUNDAY, NOVEMBER 11, 2007 
3:00 PM - 6:00 PM Conference Registration Open Grand Mesa Foyer 


MONDAY, NOVEMBER 12, 2007 
The Tutorial sessions require a $275 registration fee which is in addition to the Conference registration fee. 


7:00 AM - 7:00 PM Conference Registration Open Grand Mesa Foyer 

7:00 AM - 8:00 AM Continental Breakfast Grand Mesa Foyer 

8:00 AM - 5:30 PM Tutorial Sessions (must be registered) Refer to Following Page 
9:45 AM - 10:15 AM Break (Tutorial Attendees Only) Grand Mesa Foyer 
12:00 PM - 1:00 PM Lunch (Tutorial Attendees Only) Grand Mesa ABC Corridor 
2:45 PM - 3:15 PM Break (Tutorial Attendees Only) Grand Mesa Foyer 

5:30 PM - 7:00 PM Reception (Open to all Attendees) Atrium Display Area 
TUESDAY, NOVEMBER 13, 2007 

7:15 AM - 7:00 PM Conference Registration Open Grand Mesa Foyer 

7:15 AM - 8:15 AM Continental Breakfast Grand Mesa Foyer 

8:15 AM - 8:30 AM Welcome & Opening Remarks Grand Mesa DEF 


¢ Mr. Sam Campagna, Director, Operations, NDIA 
¢ Mr. Bob Rassa, Director, Systems Support, Raytheon Company 
8:30 AM - 9:15 AM State of CMUMI® Grand Mesa DEF 
¢ Mr. Bob Rassa, Director, Systems Support, Raytheon Company 
¢ Mr. Clyde Chittister, Chief Operating Officer, SEI 


9:15 AM - 10:00 AM CMMI@® Into the Future Grand Mesa DEF 
¢ Mr. Bob Rassa, Director, Systems Support, Raytheon Company 
10:00 AM - 10:15 AM Break Grand Mesa Foyer 
10:15 AM - 11:45 AM Executive Panel Grand Mesa DEF 
Moderator: 
Mr. Bob Rassa, Raytheon Company 
Panelists: 


Ms. Kristen Baldwin, Office of the Secretary of Defense 
Mr. Tom Neff, Defense Threat Reduction Agency 

Mr. Rich Frost, General Motors 

Mr. Mike Phillips, Software Engineering Institute 


12:00 PM - 1:30 PM Lunch with Guest Speaker Grand Mesa ABC Corridor 
¢ Mr. Mark Schaffer, Director, Systems & Software Engineering, OSD (AT&L) 

1:30 PM - 5:00 PM Technical Sessions Refer to Following Pages 

3:00 PM - 3:30 PM Break Grand Mesa Foyer 

5:00 PM - 6:30 PM CMMI-ACQ Rollout Reception Atrium Display Area 

WEDNESDAY, NOVEMBER 14, 2007 

7:15 AM - 5:00 PM Conference Registration Open Grand Mesa Foyer 

7:15 AM - 8:15 AM Continental Breakfast Grand Mesa Foyer 

8:15 AM - 11:45 AM Technical Sessions Refer to Following Pages 

9:45 AM - 10:15AM Break Grand Mesa Foyer 

12:00 PM - 1:30 PM Lunch with Guest Speaker Grand Mesa ABC Corridor 
¢ Ms. Mary Poppendieck, President, Poppendieck, LLC 

1:30 PM - 5:00 PM Technical Sessions Refer to Following Pages 

3:00 PM - 3:30 PM Break Grand Mesa Foyer 

THURSDAY, NOVEMBER 15, 2007 

7:15 AM - 5:00 PM Conference Registration Open Grand Mesa Foyer 

7:15 AM - 8:15 AM Continental Breakfast Grand Mesa Foyer 

8:15 AM - 11:45 AM Technical Sessions Refer to Following Pages 

9:45 AM - 10:15AM Break Grand Mesa Foyer 

12:00 PM - 1:30 PM Lunch and Award Presentation Grand Mesa ABC Corridor 

1:30 PM - 5:00 PM Technical Sessions Refer to Following Pages 


3:00 PM - 3:30 PM Break Grand Mesa Foyer 
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"4 = There is a marked difference between organizations 

— that truly want to implement CMMI , and those who are 

- simply try to get a “certificate” 

= Contenders invest time and energy on understanding 
the industry best practices in the model, fitting them to 
their projects and organization, and improving their 
effectiveness and efficiency 


= Pretenders simply do enough to convince an appraiser 
to give them the maturity level -- along the way, they 
de-motivate their staff with bureaucratic processes, 
disappoint their customers with inconsistent 
performance, and generally give the model a bad name. 


ASo 
Sas 
Lae ots sg 
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"4 " This presentation will take a critical look at the 
ee / difference in how contenders and presenters approach 
4 improvement 


Z = The observations are useful to 


= Customers trying to decide whether a potential 
contractor is a pretender or contender 


= Well-meaning organizations which may be inadvertently 
encouraging pretender behaviors. 
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= Goal setting 


= Training and awareness 
= Customer engagement 
= Organizational support 
"= Appraisals 
NORTHROP GRUMMAN 
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Oal 


Contenders Pretenders 

= Set realistic improvement = Generate a slogan “Level X in 
goals after an initial appraisal, 2 ”, without any idea or 
based on business goals and what it will take to get there, 
an investment return strategy or how much senior 

ue management is willing to 

= All projects participate in the investment 
improvement effort — “if it s 
good for anybody, it s good = In fact, senior management 
for everybody”. may not be willing to invest 


anything, encouraging the 
staff to “improve on their own 
time” 


= When senior management 
realizes the goal is too 
aggressive, they limit the 
improvements to a few 
—  “sherry-picked” projects. 


\ NORTHROP GRUMMAN 
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Following some version of the Learning 
IDEALS” model provides Se 

discipline to ensure CMMI - Rev 
based improvements support 
organizational business goals 


Shih Tor setcontext = Establish Plan sre 
impowenmen andestablth improvement eecule 
Sporeorhip infrastructure pilots 
Initiati er 
nitiatin Appraise and} Soe cute, 
g , characte rane andtrack | 
Gur rent \ installation 
pr echice F 
ae Establish process 
Of adionteare — 
recommendations — a 


anddocument = Bian actions 
‘Set priorities = Fan actions 
phase reasuks and strategies 
Diagnosing 7 
“IDEAL: A User's Guide for Software Process 
Improvement ,” Robert McFeeley, Software 
Engineering Institute, CMU/SEI-96-HB-001 NORTHROP GRUMMAN 
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Tr sin ss_ oals 


P Shortened time to market 
= Promotion of growth 


= Improved product quality 


Reduced development costs 


= Readiness for business change 


Achieving a maturity level may reflect 


a business goal to remain competitive, 
but focusing on that sends the wrong message 


NORTHROP GRUMMAN 
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n rlilin srintlso I 


é . Process discipline leads to predictable project performance 
— = Say what you do; do what you say 
: = Document the plans/processes 
“a = Communicate them to the performers and stakeholders 
= Audit to ensure we are following them 


2. Conscious choices lead to better processes 


=" E.g., identify relevant stakeholders and their involvement; identify 
work products to be controlled and the control method; define 
validation procedures and criteria, ... 


Organizational learning improves project performance 

= Capture what works, and what doesn’t 
| = Make rules (policies) to guide projects 
4 = Define expected processes, and let projects tailor them to fit 
; = Capture work products and measures, and learn from them 


Reference: “Interpreting the CMMI: It Depends!”, R, Hefner and 
S. Yellayi, 2005 CMMI Technology Conference and User Group NORTHROP GRUMMAN 
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*4 = Each practice provides value in possible ways 


aad, = Performance — the practice directly reduces cost and or 
ge schedule through either increased efficiency, increased 
rd effectiveness, or lowered rework 


= Quality — the practice produces higher quality products, 
by either preventing or uncovering defects 


= Communications — the practice helps everyone 
understand expected behavior, or provides insight 
leading to better decisions 


= Many practices effect more than one dimension 
= Some practices provide the potential for a positive 
impact or reduce the risk of a negative impact 


Rick Hefner, “How to Explain the Value of Every CMMI Practice” 
2007 CMMI Technology Conference and User Group, Wed, 11:00am WORTHROP_ GRUMMAN 
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Trainin an ar n ss 


_ #; Contenders Pretenders 

m ' = Realize that improvement = Simply tell the staff to do it, 
 /-_, means learning how to “because the CMMI says so” 
4 implement new practices, and 


training is key 


= Invest heavily in explaining 
how to implement the new 
practices, and why they add 
value 


Tn, 


; += 


10 
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trat iS or r ani ational Trainin 


= Start by defining the key job functions in 


— - the organization ec 
J = E.g., project manager, software engineer, 
ee quality assurance specialist 
Fd :- Identify the requisite knowledge associated 


with each function Ree 


= Define a set of course modules that impart 
this knowledge 


= Map modules to job functions 
= Some modules will be common to multiple job functions 


= Acquire training materials and trainers 
= Should reflect the organization’s policies and processes 
= Unlikely that standard vendor/university courses will fit 


= Ensure all the CMMI process areas are addressed 
= Knowledge needed to perform the process, NOT a course about 
the CMMI® requirements for that process area 
= Include performers of the process, and supporters of the process 


Rick Hefner and Sree Yellayi, “What the CMMI Doesn't Say About Training NORTISROP GFUSPIMAN 


11 (But Should!) ” 2005 CMMI Technology Conference and User Group 
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trat iS or r ani ational Trainin 


= Identify each employee by their job function(s), map to required 
—" courses 


= Ifthe employee already has the identified minimum knowledge, 
they do not need to take the course 


" Establish student records 
= Who has completed what course, waivers 


= Review required training with employees 
= Career-planning, promotions, new hires 


= Where additional project-specific training 
is required (e.g., tools, methods), adopt a 
similar approach at the project level 
= Project Planning SP 2.5 
addresses project specific training 


NORTHROP GRUMMAN 
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Oo ni ations st r sst n ri in 


= Sponsors and performers must have a strong vision of 
the desired culture 


= What are my roles and responsibilities? 

= What changes in behavior are required? 

= What are the underlying beliefs and values? 
= How do | benefit — WIIFM? 


Covert level 


Ethics alues Norms 


Intermediate level 


Beliefs Priorities 


Attitudes 


Overt level Opinions Behavior Conduct Do Donts 


Rick Hefner, “Sustaining CMMI Compliance,” 2006 CMMI Technology NORTHROP GRUMMAN 
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Rick Hefner, “Sustaining CMMI Compliance,” 2006 CMMI Technology 
14 Conference and User Group 


Achieving and maintaining mature processes . 
is essential to meeting our business goals | 
= Predictability, performance, quality yy: 


CMMI involves short-term investment for long-term gain 


CMMI is an enabler (not a guarantee) of project success 


= Other aspects (people, technology, customer relationship, etc.) are 
equally important 


= The value is often risk reduction (which may be difficult to measure) 


CMMI is aa set of proven, industry best-practices 
= Adoption is about learning how to apply these practices to our work 


= The practices may feel awkward and have limited value until we 
learn them 


= |t’s OK to make mistakes — we will get better over time 


When the entire organization is behaving maturely, everyone s job 
becomes easier 


Continuous improvement is a way of life 
NORTHROP GRUMMAN 
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Contenders Pretenders 
Know that process = Attempt to fly “under the 
improvement is about short- radar” 
in investment for long-term 5 Wihandhe oxi burdeaot 

work starts to impact project 

Engage with their customers, schedules and budgets, 
by explaining the benefits practitioners often blame 
CMMI will provide, the “that CMMI stuff” for their 
timeline needed to see those inability to meet project 
benefits, and the objectives 


improvements to be taken 


Enlist their customer s help in 
identifying issues that CMMI 
can address 


NORTHROP GRUMMAN 
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Organizational 
process performance 


= Problem behaviers are 


uantitative project 


management recognized faster, enabling 


quicker resolution 


Organizational 
innovation and 
deployment proven o 


= The project fixes the source y 
Causal analysis of defects to prevent future “ 
defects 7 | 


triro tsan ris ro ast ron ar 


Rick Hefner, “How Does High Maturity Benefit the Customer?,” Systems 


& Software Technology Conference, 18-22 April 2005 NORTHROP GRUMMAN 
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r ani ational 


Contenders Pretenders 
= Fully support the CMMI®- = Largely ignore organizational 
based improvement program support, often to save money 


by providing training, 
templates, tools, process 
assets libraries, measurement 
repositories and other work 
aids focused on improving 
the ability of practitioners to 
competently adopt the model 


= Where required by the model, 
they establish process asset 
libraries and measurement 
repositories, but they are 
largely shelfware 


NORTHROP GRUMMAN 
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ana nt o it ntan ort 


*4 =" Understands the key messages 
= Is willing to take actions to reinforce them 


_ ® Provides resources to support sustain process 
improvement efforts 


"= Sets expectations that essential project functions will 
be funded and processes will be followed 


= Project planning, estimation, tailoring, CM, QA, etc. 


= Supports process improvement and sustainment, 
rather than passing appraisals 


= Rewards mature processes development and 
sustainment rather than individual heroics 


= Tell me how you will reward me, and I'll tell how | will 
behave 


Rick Hefner, “Sustaining CMMI Compliance,” 2006 CMMI Technology NORTHROP GRUMMAN 
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raisals 


Contenders Pretenders 


= jew appraisals as money = Often only adopt CMM ona 
well-spent in measuring the subset of their projects, also 
maturity of their processes, severely limit the percentage 
identifying improvements, of projects that they appraise 


siivalciasacic alee Mk = Scope the work to what they 
= Often appraise other aspects think will pass, not what is 
than simple model best for the business. 
compliance, such as process 
effectiveness and efficiency 


= Ensure that all projects within 
the organization participate in 
the appraisal, to reinforce the 
use of mature processes, and 
to ensure all customers 
receive the benefits of that 
maturity 


NORTHROP GRUMMAN 
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raisal Ina ra is 


= Methodology 
— = SCAMPIS” A appraisals provide highly accurate appraisal results 
* = SCAMPIS” B, C, and other appraisal methods may be useful, but 
, they are not designed to provide the same accuracy 
" Appraiser skill 
= There is wide variation in appraiser skill, experience and insight 


= Although appraisal experience is a crucial contributor to accuracy, 
the appraisal methods do little to ensure sufficient experience — 
method, type of organization 

= There is wide variation in how the model is interpreted, even 
among experienced lead appraisers 


= Appraiser independence 
i = Appraiser independence is needed to ensure unbiased results 
; = It is difficult to establish a completely independent situation 


Rick Hefner, “How Does High Maturity Benefit the Customer?,” Systems 
& Software Technology Conference, 18-22 April 2005 NORTHROP GRUMMAN 
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"4 = There is a marked difference between organizations 
ee that truly want to implement CMMI , and those who are 
4 simply try to get a “certificate” 


y = By discussing the differences, we hope to help the 
CMMI community the true value of CMMI 


CMMI® ts registered in the U.S. Patent and Trademark Office by Carnegie 
Mellon University 
SCAMPIS and IDEALS” are service marks of Carnegie Mellon University. 
NORTHROP GRUMMAN 
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"4 = Convincing someone to implement a CMMI practice is 
all especially difficult when they don t believe the practice 
| adds any value 


= It is important that CMMI advocates can clearly explain 
the business value of each practice and its positive 
impact on project and organization performance 


= This presentation will outline the basic benefits 
possible from adopting a CMMI practice 


NORTHROP GRUMMAN 
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at Is I 


*4 = The CMMI is a model of industry best-practices for 
ell engineering products 


= When an organization decides to adopt CMMI, they 
commit to performing these best-practices 


= Different than a customer-driven process, where you 
simply do what the customer asks you to do 


=" ou are performing practices in what you understand 
to be the best way known in industry 
= “Best” implies predictably producing products of 
acceptable quality at the lowest possible cost and 
schedule 


NORTHROP GRUMMAN 
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o tin t I 


CMMI practices 


nig y 
g . ‘ Already Not performing 
ag performing i 
Not aware of Aware of 


Determine how 
best to perform = <————- 
and document 


Perceive as Don t perceive as 
valuable valuable 


NORTHROP GRUMMAN 
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T talInt ran 


CMMI Change Agent CMMI Change “Target” 


“ou re not doing practice X”’ “So.” 
“ou must do that practice “The customer didn t say 
to satisfy CMMI” we have to do practice X” 
“Practice X adds value” “ ow 
“Well, it s in the CMMI, “Practice X doesn t make sense 
so it must be important” for us — we re special” 
“Well..., you have to do the practice ‘ e 


. . 393 
or you II fail the appraisal NORTHROP GRUMMAN 
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n rlilin srintlso I 


Process discipline leads to predictable project performance 
= Say what you do; do what you say 

= Document the plans/processes 

= Communicate them to the performers and stakeholders 

= Audit to ensure we are following them 


Conscious choices lead to better processes 


= E.g., identify relevant stakeholders and their involvement; identify 
work products to be controlled and the control method; define 
validation procedures and criteria, ... 


Organizational learning improves project performance 

= Capture what works, and what doesn’t 

Make rules (policies) to guide projects 

= Define expected processes, and let projects tailor them to fit 
= Capture work products and measures, and learn from them 


Reference: “Interpreting the CMMI: It Depends!”, R, Hefner and 
S. Yellayi, 2005 CMMI Technology Conference and User Group NORTHROP GRUMMAN 
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Oo ot I rati s al 


‘4 ™ Each practice provides value in possible ways 


= Performance — the practice directly reduces cost and or 
Pa schedule through either increased efficiency, increased 

x effectiveness, or lowered rework 

’ =" Quality — the practice produces higher quality products, 
by either preventing or uncovering defects 


= Communications — the practice helps everyone 


understand expected behavior, or provides insight 
leading to better decisions 


= Many practices effect more than one dimension 


= Some practices provide the potential for a positive 
impact or reduce the risk of a negative impact 


NORTHROP GRUMMAN 
7 Rick Hefner, "How to Explain the Value of Every CMMI Practice", 20 
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Oo I ras r or ot ntial 


"4 = The activities which drive cost and schedule the most 
—, provide the most potential for productivity 
j improvement 


= For most large software companies and large 
software projects, the most expensive and time 
consuming activities, in rank order are 


= Defect removal 

= Producing documents 

= Meetings and communications 
= Coding 

= Project management 


Reference: “The Schedules, Costs, and Value of Software Process Improvements,” Caper 
Jones, 2007, used with permission 


NORTHROP GRUMMAN 
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arri rs to in t al 


J “Sometimes you have to believe it to see it.” 


4 = Practitioners may not have worked in an environment 
_ where the practice was performed 


= Practitioners may have worked in an environment 
where the practice was performed poorly or ina 
non-value-added manner 


= The practice may run counter to a long-held belief 


=" Believing the practice is an improvement may require 
an action the practitioner is not willing to take 


= Awkwardness of doing something new 

_ = Admit they’ve been doing it wrong 

= Loss of personal power when perceived to be an expert 
in the current approach 


NORTHROP GRUMMAN 
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r ssin t n_ rl in li s 


*4 = Sponsors and performers must have a strong vision of 
+ the desired culture 


= What are my roles and responsibilities? 

= What changes in behavior are required? 
What are the underlying beliefs and values? 
How do | benefit — WIIFM? 


Covert level Culture 


Ethics alues Norms 


Intermediate level 


Beliefs Priorities 


Attitudes 


Overt level Opinions Behavior Conduct Do Donts 


Reference: “Sustaining CMMI Compliance,” Rick Hefner, 2006 
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nts 


"Crossing The Chasm“, 
Geoffrey Moore 


(Il, EA) Explain how the practice adds value 
= Identify a situation in which this value would be realized 


(EM) Where possible, provide data 
= It is difficult to find perfect data, which “proves” the value 
= Even perfect data may not convince a skeptic 


(LM) Identify others who have seen the (qualitative) value of this 
practice 


(LM) Couch the value in ways that appeal to the practitioner 
= Reduced risk. addressing a customer issue, reduced workload, etc. 


(LM) Show ways that the practice can be done easily 
(LM) Suggest the practitioner try it and see NORTHROP GRUMMAN 
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Oo I ras r t al 
Isn t io sto an_ée ra tition rs 


‘4 ™ Planning (PP, SP 2.2) 

: = Supplier Agreement Management (SAM) 

- . . Stakeholder Identification and Management (GP 2. ) 
= Manage Configurations (GP 2. ) 

=" Risk Management 

= Decision Analysis Resolution 


= Level 


NORTHROP GRUMMAN 
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= The value of planning is communication 
= Plans provide a structured way to investigate 
the best approach for satisfying given objectives 
= Practitioners know what is expected of them, 
so fewer mistakes are made, avoiding rework 


= Interdependences are identified and negotiated which causes 
fewer disconnects between groups, avoiding schedule delays and 
wasted resources 


= We monitor progress against plans, so that corrective actions can 
be taken more quickly and insightfully 


= To realize the value of planning 
= The environment has to be predictable (vs. agile “home-ground”) 
= Plans are reasonable 
= Plans are kept up-to-date 
= Plans are communicated to those involved and followed 
= The process behind the plans is communicated and followed 


NORTHROP GRUMMAN 
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*Z4 Project Planning - Specific Practice .2 


Establish and maintain estimates of the attributes of the work 
products and tasks. 


Bt 


Measures of size of 
complexity that contribute to 
determining effort and cost 


= Frequently heard 
= “It's not important to estimate or track code size, it changes anyway” 
= “We don’t use attributes, we just know it’s a X week effort” 


= Explaining the value 


= Everyone uses attributes, perhaps subconsciously — we just want to 
make it conscious 


NORTHROP GRUMMAN 
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= The value of risk management is performance 
= By identifying potential risks and mitigating them, 
you prevent schedule delays caused by a problem 


= Without risk management, effectiveness is hindered because 
things outside your control un-do your good work 


= Must appreciate that some mitigation actions will be “wasted” 
because the risk is never realized 


= To realize the value of risk management 
= The culture must support an open discussion of risks 
= Must be effective at identifying ALL risks, ALL perspectives 
= Risk mitigations must offer a reasonable return on investment 


= Solid plans must be written so the risks (of not meeting the plans) 
can be determined 


Reference: “Relationship Between Risk Management and 
Project Performance, “Warren Scheinin, Wednesday, 3:30 PM 

NORTHROP GRUMMAN 
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Risk Management - Specific Practice 


Determine risk sources and categories. 


How should we categorize the 
risks once they are identified? 
¢ Urgency 
¢ Assignee 


What sources of risk should we 
consider in identifying risk? 


= Frequently heard 
= “The sources are the same as the categories” 
= “Anybody can identify a risk at any time” 
= “Risks are categorized by impact: cost, schedule, performance” 


= Explaining the value 


= Setting expectations for the types of risk to consider helps ensure 
that ALL risks get identified 


= How the risks will be categorized helps ensure they are managed 
appropriately 
NORTHROP GRUMMAN 
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I is Ta 


A. Product Engineering 


. Requirements 


a. Stability 

b. Completeness 
c.Clarity 

d. Validity 

e. Feasibility 

f. Precedent 

g. Scale 


. Design 


a. Functionality 

b. Difficulty 

c.Interfaces 

d. Performance 

e. Testability 

f. Hardware 

g. Non-Developmental SW 


. Code and Unit Test 


a. Feasibility 
b. Testing 
c.Coding/Implementation 


. Integration and Test 


a. Environment 
b. Product 
c.System 


. Engineering Specialties 


a. Maintainability 
b. Reliability 
c.Safety 

d. Security 

e. Human Factors 
f. Specifications 


ono 


B. Development Environment 


. Development Process 
a. Formaility 
b. Suitability 
c.Process Control 
d. Familiarity 
e. Product Control 

2. Development System 

a. Capacity 
b. Suitability 
c.Usability 
d. Familiarity 
e. Reliability 
f. System Support 
g. Deliverability 

. Management Process 
a. Planning 
b. Project Organization 
c.Management Experience 
d. Program Interfaces 

. Management Methods 
a. Monitoring 
b. Personnel Management 
c.Quality Assurance 
d. Configuration Mgmt 

. Work Environment 
a. Quality Attitude 
b. Cooperation 
c.Communication 
d. Morale 


C. Program Constraints 


. Resources 


a. Schedule 
b. Staff 
c.Budget 

d. Facilities 


. Contract 


a. Type of Contract 
b. Restrictions 
c.Dependencies 


. Program Interfaces 


a. Customer 

b. Associate Contractors 
c.Subcontractors 

d. Prime Contractor 

e. Corporate Management 
f. Vendors 

g. Politics 


Reference: “Taxonomy-Based Risk 
Identification,” Marvin J. Carr, et al, 
Software Engineering Institute, 
CMU/SEI-93-TR-6, ESC-TR-93-183, 
June 1993. 
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0 


= The value of Level is communication A\ <X 
and quality > 
= Level 4/5 is 10-20% cheaper than Level 3, 7 


even though more is being done 
= Quantitative management establishes expected ranges of process 
performance 


= Process are stable and predictable — allows un-usual process 
behaviors to be quickly identified, so that effective corrective action 
can be taken 


= To realize the value of Level 
= Processes have to be stable (performed consistently) 
= Processes under statistical control must support business objectives 
= Data has to be useful and clean 
= Analysis has to lead to actions 


NORTHROP GRUMMAN 
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antitati ana nt 


= Suppose your project conducted 
—7 several peer reviews of similar code, 
2 and analyzed the results 


: = Mean = 7.8 defects/KSLOC 
a = +30 = 11.60 defects/KSLOC 
) = -30 = 4.001 defects/KSLOC 


UCL=11.60 
11 
10 
£ 
= 2 
aa 8 
To) Mean=7.8 
| 
O 7 
= 
om 6 
= 
5 
4 LCL=4.001 
3 


0 5 10 15 
Observation Number 


What would you 
expect the next peer 
review to produce in 
terms of defects 
KSLOC 


What would you 
think if a review 
resulted in 

defects KSLOC 


defects KSLOC 


Reference: “A Practical Guide to 
Implementing Levels 4 and 5 (tutorial),” 
R. Hefner, 2006 CMMI Technology 
Conference and User Group 
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"4 = Convincing someone to implement a CMMI practice is 
—/ easier when you can explain how the CMMI practices 
& add value 


cd ) = Insight comes from observing practical, value-added 
| implementations 
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Z pars and Eventual Acceptance of CMMI 


e Research Question 
e Setting/Method 

e Analysis 

e Conclusions 
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e Original Research Question 


—- Organization Theory 
e Formal organization — codified process, org. chart, etc. 
e Informal organization — unwritten, cultural, normative 
—- Responsive to importance and uncertainty 
e Systems of control 
— Direction 
- Evaluation 
- Correction 


e CMMI as formalization of process previously controlled by 
informal organization 


e Initial reactions 
e Follow-up 3 Years Later 
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e Setting/Method 


— Setting 
e Defense contracting, complex embedded 
Submarine systems, New England 
e ~1000 Employees 
e >70% professional (engineers/managers) 


— Method 


e Ethnographic, grounded theory, participant 
observation 
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e Analysis 
— Specific Phase I patterns 
e Production of artifacts 


e Assumption of a perfect product 
e Performance evaluation and protection 
e Promotion of self-interest 
e Vertical migration of task definition 
—- Phase II 
e Strength of informal organization (culture/structure) 
e Strong norms regarding customer expectations and 
product quality 


e Intelligent integration of CMMI with other processes 
(peer review databases, development tools [DOORS]) 
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e Conclusions 


- Evaluative facet of CMMI process 
initially threatens to erode community 
and hinder productivity 


—If informal organization is effective 
and strong, these negative effects are 
avoided and CMMI’s performance 
enhancing features, as a tool, are 
embraced 
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Production of Artifacts 
PHASE | 


“It did not change how we develop software. It changed the artifacts 
that are produced, it changes individual accountability for producing 
artifacts. It drives up the costs in terms of the number of hours you 
have to expend- that cost has to be absorbed by the individual 
developers because costs can only go down - never up. The rationale 
is that this is going to make us cheaper and to do that -— and they really 
just put more pressure on individuals to follow more religion, produce 
more artifacts — so what it comes down to is software always 
developed the same way and you can tell that by watching a project. “ 


PHASE II 


felt while some of the “overhead” is undesirable and takes away from the 
“real work,” by and large this “tax” was not a large burden. Furthermore 
they felt that it did not sap the effectiveness of the informal process. 
Engineers still consulted with one another and performed informal reviews as 


they had done in the past. Incorporation into DOORS and peer review tools 
minimized overhead. 
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Assumption of a Perfect Product 


PHASE | 


“So you go into a room with a work product, and anything that you say is 
wrong with it is considered a defect. That means that if somebody feel that if 
it’s appropriate, there’ll be a root-cause analysis. “Why did you do this?” So 
you replace the useful engineering discussion -— a work group with something 
that is a lot uglier. You know, it’s based on the assumption that there 
shouldn’t have been anything wrong in there. The point of the process is to 
improve the product. There’s the expectation now that the product will go in 
there near perfect.” 


PHASE I! 


As CMMI processes became institutionalized and accommodated by the 
informal processes, it became clear that informal pre-reviews largely 
mitigated these negative effects by being positive, non-critical, and 


constructive. 


Your complimentary 
use period has ended. 


£ Co m 0 | ete Thank you for using 


PDF Complete. 
pars and Eventual Acceptance of CMMI 


peered Features 


Performance Evaluation and Protection 
PHASE | 


“Although everybody’s assured this doesn’t happen, and it’s 
denied vehemently, people know that management goes into 
these databases and looks for individual producers and defects 
as part of how to establish who their high performers are — who 
produces a quality product. That makes people very defensive 
when they go into these forums.” 


PHASE I! 


“I try to call everything minor. Because if you flag something as 
a major, if somebody’s looking at that database that can be 
more of a mark against an individual. Again, from a Systems 
point of view — what discriminates the two, there’s a definition 
in the process as to what the severity is, but ultimately it’s a 
subjective call. You try to justify something being a minor 
however you can. Nobody likes to say there’s a major defect in 
somebody’s work product.” 
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Promotion of Self-Interest 


PHASE | 


“If you have a milestone to meet. If you have an artifact to produce and regardless of where your 
true commitment falls, when somebody draws a line in the sand and says you have to get 
something done by a particular point in time and then there’s something else that really should be 
done by a particular point in time but doesn’t directly reflect on that artifact, where do you place 
the priority? You prioritize based on what you’re being measured to when push comes to shove. 
If what you really needed to do was take the time to work with somebody else to work out a 
critical interface, ... let me back up. It promotes more self-interest. If individuals are being 
measured with milestones rather than measuring the crew, you know, based on what they really 
accomplish, people are going to act individually. It forces that reaction. You can’t survive in that 
type of environment if you don’t accomplish what you're individually accountable for. So 
everybody to some extent strives for a balance between the two. There are some people that try 
to be completely rebellious to that and those people end up well -— they’re outside of the 
organization after a short period of time.” 


PHASE I! 


The phase II data showed no increase in self-interested behavior, although the 
data gathered were not quantitative. The author attributes this to the more basic 
finding that the informal community/ organization at the heart of the systems 
engineering tasks maintained its culture and structure and therefore norms 
dictating self-interested versus altruistic behavior were not changed. 
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Vertical Migration of Task Definition 
PHASE | 


“The other observation that I think is worth looking at is that this could have a detrimental 
impact on the capabilities of this organization. And maybe this is what separates the notion of 
the hero from somebody else. But the more they’re building a mentality that developing a 
product is turning a crank — executing to the process — the more individuals are going to rely on 
being spoonfed. The design of a system is really nothing more than working with different 
levels of abstraction at different points in time. For example, when you do lay something out at 
a high level you rely on individual software engineers — the people responsible for something, to 
solve significant portions of the their own problem - to finish coloring in the details on their 
interface. The more people expect something to be a matter of turning the crank, the more they 
look to other people to have something defined for them. So that taps the resources to go down 
to an unreasonable level of detail. To go down multiple layers of abstraction, beyond where 
they really should be to solve other people’s problems because the organization’s becoming less 
capable.” 


PHASE Il! 


Since, from the point of view of the customer, the “deliverables” did not need to 
carry with them CMMI artifacts, a strong pattern emerged in which the technical 
content of the product was the highest priority and if time ran out, the formal peer 
reviews and artifact gathering was postponed until after the official delivery. 
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A. Campus Management is not: 


An aerospace giant — developing software to design jet engines 
or weapon systems 


A medium size firm who needs to be a specific CMMI maturity 
level to compete for defense contracts 

A telecommunications leader writing software for the 
10,000,000 cell phones it sells per year 


A. Campus Management is 


A privately held software firm started approx 17 years ago with a 
vision to provide business solutions to the “for-profit” segment of 
the secondary education market 


Approx 300 employees — growing rapidly 
Has expanded its customer base to the “not-for-profit” segment 


Revenues and success 100% dependent on the quality of its 
software 


Working hard to achieve CMMI Maturity Level 3 in the 2Q 2008 3 


CAMPUS’ 


MANAGEMENT 


Your complimentary | 


ee use period has ended. 
Complete "a ; 
Talie)daarctice)aM anal mam er-)ai tela 


paeemed Features 


| | 
= aria f 
£ Li | 
Alumni Association Tk 
2S ie = 
. g | 3 g Development 
Finance 
4 


Registrar 


MANAGEMENT 


Your complimentary 
use period has ended. 
Thank you for using 


PDT 


¥ Complete 


PDF Complete. 


fee Features 


CAMPUS UNIVERSITY 


MAH AGEMEM T 


(oe) sy ib(e(salmacelars! 


Hello Moma! Logout | Helo 


wi Some Page 


- Campus Info. 


, Oo 


Fane ap fication 


- My Academics 


. Transfer Credits 


a bal Housing 


7 My Profile 
waiigesone Cepber. 
Fs My Finances 

- My Financial Aid 
aio cements. 

- My Career 


- External Links 


Weather 


Comics For foc 
fetter, Fl at os 5o an 
EsSr 


Current Conditions: 
Light Drizzle, 61 F 


Forecast: 

Tues - Sprinkles, High: F3 
Low; GL 

Wed - Partlh Cloudy. 
High: 78 Low: 66 


Full Forecast at Yoho! 


Ayte ather 
(provided by The 
Weather Channel) 


Advisor Center  —|*) 


NEW State roup 
pee | Mani Mehta 


—|%) 


News Center 


News 


reece pr 


Camous Management Corp 
Snnsunces new contigurable web 
portals, oroviding schools with more 
contrall...more 


Corre Bryan of Highland, Md has 


received an Associate of Arts deqres 
in Culinary 4rts....more 


Calendaronav 


My Calendar 


Tuesday 


Choose Day Ea 


February 7007 


LS =i = 


Febritany 06, 2007 


MSN TopStories =|*) 
445, ue wanes fires tee ie 
Saghiatad 

The 4mericean snd Ireqi olan to 
pacity Baghdad rests on the 
acsumpton that U.5. troops can 
win the trust of Ss wary 

populsabon by pratecting civilians 
trapped arid sectarian warfare, 


Astra? amested in ategead 
Paratd 


4, Fo Oo 
astronaut 
charged wrth 
atbsmpbng to 
kidnap a 
romantic rival 
Ina lave 
trianole with another astronaut 
wes allamwed to go free an bail 
Tuesday on the condition chat 
she not contact the alleged 
victim, 


Wowso Teen" 


4 
Inve stigaton 


Aine ibestigators” 


CAMPUS" 


MANAGEMENT 


Your complimentary 
use period has ended. 
Thank you for using 
PDF Complete. 


3 Fc ellete 


Click Here to upgrade te 
Unlimited Pages and EXpanaueeenaiaees 


Career 
Placement 


Career 
Services 


Scheduling, 
Advising & 
Degree Audit 


ys 


GL, HR, 


Payables/Purchases 


MANAGEMENT 


Marketing & 
Advancement Recruiting 
Admission 


& 


Recruiting Admissions 


Academic 
Records 


[One Campus 


Web portals and 
workflows to serve 
students better 


Alumni 


Student 
A t 
ecoeS Financial Aid 
and Scholarships 


Finance & 
Accounting 


Accounts 
& Billing 


Course 
Management 


Advising 
& Audit 


CAMPUS" 


Your complimentary 
use period has ended. 


= <PDF 
E Complete everson FT) AV ayag EV(=) Mv oO a=) 


ae an mew Features ) a g rs | n 


A 2Q 2005 — major release of software — quality concerns from 
clients expressed at annual users conference 


Periodic quality initiatives had been attempted but with no 
structure and quickly abandoned 


As delivery dates were being approached and client demands 
were increasing all existing process was thrown out the window 


Plenty of CMMI Maturity Level 1 Heroes 
Enterprise clients demanded quality improvements 


A 3Q 2005 — executive decision to commit to quality and filled 
Director of Quality position 


A 4Q 2005 - 
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CMMI —- is it the answer ? 
2005 NDIA CMMI Conference 
Consultant Preliminary Search 7 
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A 1Q 2006 


| SEPG Formation 
A Two project directors 
A Three aspiring project managers 
A Documentation specialist 
A.No CMMI training, but PMI experience 
A Influential and enthusiastic 
A Unfortunately no software developers 
A Weekly meetings 


| CMMI Consultant Selection 
A Interviewed four consulting firms 


A Selected Don Franke Associates per recommendation of a 
South Florida Lockheed division and a Pratt & Whitney 
division 


| Reorganization 
A Director of Quality - Independent of Development 
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Policies and Plans Wi} Software Release Approval Process 3/30/2007 1:57 PM © CMC Product Development Portal Pro cess 
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Repository wi} Design Review Guidelines 2/7/2007 5:35 PM « Configuration Management 
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A Internally Developed Training 
| Department wide “Introduction to CMMI’ 
| Executive Overview 
| Introduction to Six Sigma 


A External Training 
| “Introduction to CMMI V1.2 Training” for SEPG members 


A Initial Consultant visit 1Q 2006, and then every quarter 
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A Expectations — Management 


CMMI is not a magic bullet 
Maturity Level 2 Project Plan 


AExpectations — Development Staff 


Do not have to become CMMI experts 
| Controlled - gradual change 


| Weekly hallway meetings — 5 minutes max 
| Positive Reinforcement — convince they are doing a 


good job (even if they are not) © 


A Expectations — Customers 


CAMPUS’ 


MANAGEMENT 


Monthly presentations on progress 


Your complimentary | 
use period has ended. 


> PDE 
# Complete  ™izeversere Fit) Vag VC=) ve) OL a=) 


: pane” Features ‘| etri CS ; Mi et ri CS 5 M etri CS 


AWhy? 
| Not because Measurement and Analysis is a 
maturity level 2 Process area but because: 
AExecutives are impatient 


AAs soon as the CMMI Consultant leaves the room 
they start whispering, no — actually speaking out 
loud: “18 months its the average, but we are better 
than the average, we will achieve maturity level 2 
in 6 months” 


ASolution — immediately create a baseline of 
whatever quality indicator is important 
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A SCAMPI C - 
| do your own, easy to do, doesn’t have to be perfect 
| great communication reference with executives 


A SCAMPI B — critical, why 


| Not free, and if executives won't pay for one to show their 
commitment, then stop, do not pass Go 


| A great pretest for assessment team and organization 


A SCAMPI A - 
| Treat it as fun, culmination of great work 
| Do not stress out organization 


| Just like any other week - if it is not, are you really CMMI 
Maturity Level 2, or 3, etc ? 
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ATraining i word of caution i make sure internal trainer is good, 
not just good, very good | keep sessions to the point, not much fluff, 
no silly games 


AProject Planning | the key process area 


AQuality not CMMI - Donékeep talking CMMI but the message 
is quality 1 when it comes right down to it that is what it is all about 


AExperience | hire someone who has CMMI experience 


AQuick Hits 7 almost subliminal CMMI messages 


ACustomers i most important advocate ” 
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AGoogle i don't become addicted - doné want your weekly 
Status report to read f Googled Measurement and Analysiso 


AGraybeards - great source for baseline but be skeptical 


“We got some tall tales that we love to tell 
They may not be true 
But we sure do remember them well” 
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Almmediately performed a CMMI Maturity Level 3 SCAMPI C 
| Kept executive buy-in 
| Development organization realized the journey is not over 


| helped prevent regression (not talking about testing) 


A Agile Methodology 
i First reaction 1 FOh Nool and more rGoogle-ingo 
| Led to switch to consultants | David Consulting Group 


| Doing some great things with Project Planning and related 
process areas | adopting into standard life-cycle process 
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Mrs. Doubtfire Answers Your 
Questions about Process 
Improvement 


CMMI Technology Conference & User Group 
12-15 November 2007 
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Background 


- Is your organization acting “childish” 
about process improvements? 


- Do they whine and cry when they have to 
learn new practices? 


- Are you at your wits end? 


You're Pregnant - Now What? 


- For better or worse, you've been given 
the responsibility to guide your projects 
to CMMI Level X 


- Parenting your projects will be the most 
rewarding (and most frustrating) thing you 
will ever do 


- Starting talking to successful parents 
about what they did 


Sie 


Sibling Rivalry - 
Introducing the New Baby 


- Baby CMMI will compete for favor among 
other work on your projects 


- Talk to your significant other (the sponsor) 
about their expectations 
- Discipline, sharing duties, etc. 


- Plan a date night (about once a month) to 
review status against expectations 


Sie 


The Terrible Two's 


- New babies are cute, but there are lots 
of diapers to change 


- As your CMML initiative matures, expect 
some pushback 
- "No!" can be a way of testing limits 


- Set firm groundrules 


- Destructive behavior (biting, hitting, refusing to 
share artifacts and measures) must be stopped 


Establishing a Bed-Time Routine 


- Children need routine to feel safe 
- Rules on acceptable and unacceptable behavior 


- Establish a routine for working together 

- What is the improvement approach and timeline? 
- What is expected of the projects? By when? 

- How will progress be measured and reported? 

- What are the consequences? 


- A daily nap often improves the attitude 


Corporal Punishment: Pros and Cons 


- You can catch more flies with honey than 
you can with vinegar 


- Firm expectations and a regular routine 
will often eliminate the need for 
punishment 


- If you must spank, you and your sponsor 
should agree on the time and place 


Temper Tantrums 


- Losing their temper is your child's way of 
dealing with the loss of control 


- Remember to stay calm, and act like the 
adult 


- Do not tolerate physical and verbal abuse 
- Use time-out when needed 


Sharing with Others 


- With multiple siblings, there is often a 
issue with sharing (artifacts, measures, 
progress, etc.) 


- Remind the projects that good families 
(organizations) share assets for the 
benefit of all 


- Giving up something now means getting something 
else when you need it 


- Maintain diligence to block access to 
dangerous items (e.g., non-compliant 
assets) 


Potty-Training Made Easy 


- Projects sometime grow dependent on your 
help for cleaning up the “dirty stuff” 


- Help projects understand that maturing is 
about taking responsibility for your own 
needs, and only asking when you need help 


- Reward them when you observe the 
desired behavior 


- Let them know “mistakes” are OK 


Sie 


Separation Anxiety 


- As we approach the “big day” (appraisal), 
expect some clinging behaviors 

- What questions will they ask? 

- What should I say? 

- Can I just stay home? 


- Help them look forward to the experience 
- This is a chance to show what you can do 


- A dry-run may help relieve the anxiety 


Your Child May Never Become a 
Doctor 


- If you follow this advise, your projects 
may grow up to be healthy and happy 


- Remember that all projects are different, 
and each are special in their own way 


- Learn to love yours unconditionally, no 
matter how they turn out 


Sie 


Contact Info 
Mrs. Doubtfire can be reached through: 


Rick Hefner 
Northrop Grumman Corporation 
rick. hefner@ngc.com 
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Does the EPG write processes rather than 
define processes? 


e Who determines the process content? 


¢ How is the best organizational process 
determined? 


e Who's processes are they anyway? 
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e If work is being conducted processes are in place 


e May be pre-defined by previous experience of 
workers (education, readings, trainings, jobs) 


e May have evolved over time through doing the work 
e Often are not consistent across all employees 


e Individuals my improve but improvements not 
communicated or carried forward 


e No measure of improvement; no concept of change 
impact to follow-on process 
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No one person may be able fo say what the 
process is - maybe not even their own 


No process documents are available 


Individuals may document for personal 
consistency, quality improvement 


No notion of process in the organization 


Work is ad hoc. Process relies on individual’s 
capabllities 
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¢e Those implementing the process know the 
real process 


e The process must be extracted from the 
subject matter experts and formally defined 


e The extraction is accomplished by the 
process change group 
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e Use stakeholders throughout the process 
definition process 


¢ Stakeholder involvement is necessary for 
commitment to the process 


e Stakeholder involvement makes the process 
their own 
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SMEs help define the process 


SMEs validate definitions 


SMEs validate organizational process 


SMEs validate written process 


SMEs are the process owners 
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e Interview subject matter experts to define every 
variation of the process 


e Create a process model of each variation (flow 
chart) 


e Perform stakeholder reviews to validate process 
model 


e Analyze the variations for commonality and 
‘legitimate’ variation 


e Model the organizational process based on the 
commonality; determine criteria for variations 


e Perform stakeholder review of organizational process 
model for commitment 
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e Create a written process based on the 
organizational process model 


¢ Stakeholders review the organizational 
written process 


¢ Create written process variations for tailoring 
purposes 


Note: No process improvement going on 
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¢ The processes belong to the Subject Matter 
Experts — they know the best practice 


¢ The EPGs role is to 
— Assist the SMEs in developing a written process 


— Coordinating the development of the 
organizational process 


— Gaining workers commitment to the 
organizational process 


— Defining tailoring variations with criteria 
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e A written process that matches the work 
being performed 


¢ Commitment to follow the written process 


e A process baseline that allows real process 
improvement 
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e Processes belong to stakeholders/SMEs 


¢ The EPGs role is to document the real process 
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Developing a Second 
Generation Directive System 
Architecture 


Ken Weinberg 
El Segundo, CA 
kiweinberg@raytheon.com 


November 14, 2007 
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meget 2) Features 


ALegacy directive systems 

ALimitations and Drivers for Change 

AProposed Solutions 

ANew Directive System 

ANew System for Non-directive Guidelines and Templates 
ALessons Learned 
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2ring Process 
Management Commitment 


a High Level 
Directive/ Directly Traceable to 
Non-Tailorable CMMI, ISO, Corp Stds 


Lower Level, 
Directive/ Further Direction on 


Tailorable rHowoto 
Meet Requirement 


Guidelines/ 


Non-Directive 
Templates 
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Aengineering only | other areas (e.g., HR) only have policies 
1 Systems are not compatible 
ATwo levels of directives i Procedures and Work Instructions 


1 Programs have trouble keeping track of multiple directives per process 
area 


ANon-Directive enablers in directive system 

| Confusing during audits 

1 Adds unnecessary formality to many tools and templates 
ANot well integrated with corporate directives 

1 Developed prior to corporate directives becoming CMMI compliant 
ALarge number of documents to manage (566) 


Time for a change 
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Old Process New Process Major Categories of 


Policy Policies Management Commitment 


Replace with Shows Traceability 
Traceability Database to CMMI, ISO, Corp Stds 


Single Level 
Tailorable Directive 


Remove from New Enabler 
Directive system System 
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seer i Features 


ACompatible with other (less complex) Corporate Directive 
Systems 
ASingle level of directives 
1 All Program Requirements in Work Instructions 
1 Less complexity 7 Keep short and concise 
ANon-Directive enablers outside of directive system 
| Eases maintenance 
ABetter integration with corporate directives, Industry 
Standards and Local Standards 
| Traceability Database 
AReduced number of documents to manage (152) 
1 Removal of 414 Enablers 
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A Transfer Functional DisciplinesdEnablers to local 
control 


A process simplification 
Areduce bureaucracy 
Areduce costs 
Aimprove cycle time 


More Useful Enablers 
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feed Features 


Type 


Control Repository Function/ Informal 
Control or Organization/ 
Local CCB Corp. Level CCB 


Repository Docushare Docushare Clearcase 
Local PAL Local PAL Corp. PAL 
Delivery Repository Web Page EMS User Views 
Search (Functional, Corp. Process 
Program, Dept, 
Site) EMS User 
Views 
Relationship to | None Expected Could be 1 
Directives if related to WI 
Requirement 


Three Levels of Control Defined for Enabler Categories 
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Bureaucracy Reduction 


Raytheon 


New 
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AHeaded Toward Single Directive System 

ASingle Level of Management Commitment 
ASingle Level of Directives 

ASimplified System - Keep it Short and Simple! 
AReduction in Directive System Size (414 Enablers) 
AeEnhanced Enablers in Separate System 

AChange Takes Time 
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Scientific Breakthroughs in Process Improvement 


Change Delivery Group 


Cheryl White 


CMMI Users Group: NDIA 
November 14, 2007 


Change Delivery Group 


“Everyone nods in 
agreement, but when it 
comes to actually changing 
processes, all we get is 
change resistance and 
work arounds." 


“We implemented the 
software the business 
says it wants. Why won't 
the business use the 
systems?" 


Sound Familiar? 


"I know what the 
problem is with the 
process. I can explain it 
in one sentence. Why 
does it take so long to 
fix it?" 


“I want things to 
change around here and 
I want it now. Not 
tomorrow. Not next 
month. 
NOwiIII" 


© 2007 by Change Delivery Group 
All rights reserved 


“We all agree. We have 
to improve our 
processes And we will... 
when we are not so busy 
fighting fires." 


“We've posted best 
practices on the web. 
We've trained everyone. 
But our defects are still 
way too high." 


9/18/2007 


Change Delivery Group 


Conventional wisdom is 
like an old guard; it 
would rather die than 
surrender. 


R. Gharajedaghi (1999). 
Systems Thinking: Managing 
Chaos and Complexity 


PI Takes Too Long & Costs Too Much 


Each year 65%-90% of all executives plan 
major initiatives than require business 
process improvement. Historically, 85%- 
95% of these fail 


New scientific research finds that road 
blocks to BPI have little to do with 
employee willingness or ability to 
change 


They have a lot to do with how 
organizations behave as systems 


To change faster and more reliably, we 
must change the way we think about 
organizations and how we change them 
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Change Delivery Group 


Improved Processes 
plus 
Culture Change 


equals 


Sustainable Performance 
Improvements 
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Breakthrough Challenge # 1: 
hange how we think about BPI 


Change Delivery Group 


Science says process improvement is not a process; 
its is an output produced by a process 


Three steps to process improvement 


PAnalyze 
a Corporate r 


) the System 


3D SCIENCE) com 
Any sufficiently advanced technology is indistinguishable from magic. Arthur C. Clarke 
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Breakthrough Challenge # 2: 
dd New BPI tools from science 


Change Delivery Group 


Moving change from “soft skill" to “hard science" 


Tools from science supplement 
more traditional BPI approaches 
such as: 

Change management 
Psychology 

Human Factors Engineering & 
Organization Development 
Teaming, mentoring & 
empowerment 

Definition of vision, mission, & 


Science offers new, more 
reliable tools for business value 


process improvement Command & control 
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Anxious (fight, flee Calm (reasonable 
or freeze) and unafraid) 


Neuroscience is the study of how our brains work at the 
neurological level. Magnetic Resonance Imaging (MRT) 
Technology offers a look into the shadowy realm of human 
cognition 
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Neuroscience Says 


Change Delivery Group 


When the Brain Meets Change 


Emotio 


Aange hurts. Real change causes real paif? 


C. Koch (2006) “Change Management: Understanding the Science of Change” CXO Media, Inc. Referencing Weston, Kilts, Blagov, (2006). “The Neural Basis of Motivated 
Reasoning” Journal of Cognitive Neuroscience; Also Lord, Ross and Lepper (1979), “Biased Assimilation &Attitude Polarization: The Effects of Prior Theories on 
Subsequently Considered Evidence”, Journal of Personality and Social Psychology. New York Times. “Your Brain on Politics”. Nov. 12, 2007. 


8 © 2007 by Change Delivery Group 9/18/2007 
All rights reserved 


| Breakthrough From Neuroscience 


Change Delivery Group 


People adopt processes faster when new processes are 
introduced new using methods that avoid triggering pain 
centers in the brain 


9 © 2007 by Change Delivery Group 9/18/2007 
All rights reserved 


Unlimited Pages and ean ew Business Theories from Biology 


Change Delivery Group 


When people interact in groups, something supra-human emerges. 
Biology contributes insights on how groups behave, how they change, 
and the role of culture in collective survival 


10 © 2007 by Change Delivery Group 9/18/2007 
All rights reserved 


Change Delivery Group 


11 


Biology Says 


Patterns constrain people's choices, 
making them act in ways that reinforce 
those patterns 


Group are bigger than people and group 
outcomes don't reflect the desires of 
any single person 


Bad outcomes have more to do with bad 
patterns than with bad people or bad 
choices 


M. Buchanan (2007). The Social Atom. 
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BPI Breakthrough From Biology 


Change Delivery Group 


Patterns can be seen in the processes people prefer to 
use. Changing the culture rules that drive behavior 
changes choices faster than training & coaching 


To implement process 
improvements fast, use 
methods that tap into 
the collective 
intelligence of the 


group 
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General Systems Theory says 


Organizations are Self- 
Regulating Systems 


Organizations live out their own 
lives while employees come and go 


Organizations are continually self- 
regulating-- correcting the 20%- 
25% of instructions in the work 
place that are always messed up 


Error correction and change 
happens based on processes 
functioning automatically at the 
organization level 


L. Skyttner. (2007). General Systems Theory 
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Change Delivery Group 


Process improvements are implemented faster when 
process engineers use methods that apply the 
organization's natural self-regulating, error correction 
processes 


A Systems View 
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Business Theories from Complexity 


Change Delivery Group 


Complexity Theory describes how organizations form, how 
they evolve and how they remain internally stable 


At the same time they change operationally in response to 
their environment 
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Complexity Theory Says 


Change Delivery Group 


Organizations are Dynamically Adaptive Systems 


Organizations are Complex Adaptive 
Systems that dynamically adjust to 
changes in their environment 


They are designed to systematically 
adapt to, and co-evolve with 
changes in the business environment 


Self -organization causes companies 
and teams to dynamically adjust to 
counter balance changes generated by 
external pressures 


Organizations continually adapt to each other through evolution, thereby 


organizing themselves into an exquisitely tuned ecosystem. 
M. M. Waldrop (1992). Complexity 
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Change Delivery Group 


Process improvements are implemented faster when CIOs and 
change agents maintain the internal balance of the group. 


‘wis External Pressures from Po 


Business Environment 


This means using methods that piggy-back BPTI initiatives onto 
the natural self-organizing processes of the organization 
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Change Delivery Group 


Chaos Theory describes how order emerges out of apparent 
disorder and how complex behavior patterns in organizations 


emerge from one simple behavior algorithm 
9/18/2007 
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Change Delivery Group 


Organizations are Non-Linear Rules Based Systems 


Chaos Systems are modeled by plotting the 
output generated when simple, non-linear 
algorithms are performed repeatedly. 


Resulting “fractals” are often beautifully 
complex 


Similarly, one simple DNA-like algorithm is 
responsible for producing the rich variety of 
behaviors found in every organization 


Non-linear systems neither settle down into an equilibrium point nor fly 
toward infinity. They are quite stable and predictable because they 
maintain dynamic balance. £. Hunt. (2007). The Mathematics of Behavior 
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Change Delivery Group 


“DNA analysis" of an organization shows why employees behave like 
they do and why they resist process improvements 


It provides guidance on how to package and implement new 
process to minimize resistance 
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w Business Theories from Physics 


Physics describes how organizations emerge, how they process 
information, and how they can be re-programmed 
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Physics Says 


Change Delivery Group 


Organizations are Information Processing Systems 


= Errors are spontaneously introduced into organization operating 
systems resulting in instabilities that are detrimental to performance 


= Organizations automatically perform error correction processes to 
maintain stability and continuity 


The distributive nature of human information processing allows people to 
cooperate. It is the richness and complexity of our shared information 
processing that has brought us thus far. S. Lloyd (2006). Programming the Universe 
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— BPI Breakthrough from Physics 


Change Delivery Group 


Organizations are programmable systems that can be 
debugged by adding new processes and removing 
spontaneously generated process errors 


Code t:::Ghange it 
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Applying Science to BPI increases speed and reliability 
of process improvement while decreasing cost 


le] Ie pea J \ 
Caleyal=y) Level of Knowledge (expert) 


With today's new, science-based methods for transformation, 
projects that once took 42 months can now be accomplished in as 


little as 90 business days. 
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Avoid generating 
pain 


Tap into the 
collective 
intelligence to take 
advantage of herd 
behavior and hive 
mentality 


Apply the 
organizations natural 
error-correction 
processes 


> 
= 
= 
O 
< 


Identify & follow 
the organization's 
rules for change 


Maintain balance 
within the group 


Debug 
dysfunctional 
organization rules 
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Questions & Discussion 


Change Delivery Group 


The 
Beginning 
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For more information on the 
Science of Change and how to apply it in your 
project please contact us 


Change Delivery Group 


303-523-5541 


email: LauraWilcox@changeperfect.com 


Presenter's Bio 


Change Delivery Group 


Ms White has more than 20 years experience as an executive 
coach, corporate trainer and organization change consultant. 
She is Executive Partner of Change Delivery Group, a company 
that enables client teams to apply the Science of Change to 
business process improvement, culture change and other in- 
house strategic initiatives. 


Change Delivery®, our flagship product, is a packaged 
organization transformation technology based on the Science of 
Change. The methods included in this package overcome change 
resistance for least time, least cost delivery of business value. 
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The What, When, Why and How 
for CMMI Training 


AVISTA 


People. Process. Promise. 


CMMI® Technology Conference 
November 12-15, 2007 


AVISTA :: People. Process. Promise. 
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» Introduction 

» Understand the Requirements 
» Lay the Groundwork 

» Plan the Training 

» Monitor Training Effectiveness 
» Summary 


» Questions & Contact Information 


AVISTA :: People. Process. Promise. 
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AEng. 
AR 
AMgmt 
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Why is Training Critical 


Reduces 
ARetraining 
ACorrective Action 


Knowledge ]I!nstitutionalization 
AAppraisal Risk 


AVISTA :: People. Process. Promise. 
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“Train People” 
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Plan 


Direct 
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Timeframe of test? 


Biecelialiare| 
Event 


Measure Effectiveness 


Pass/fail criteria? 

Hint 1: Ask “Have you used the 
process/tool?” 

Hint 2: Question design is crucial 


WO 


Tracking SYSTEM: WhoG had what training? When was the 
training? What training method? What version of training? 
Effectiveness & evaluation data 


AVISTA :: People. Process. Promise. 


Measure Evaluation 
Hint: Document type of delivery 
(E.g. Video lecture, In-person 

lecture, Hands-on workshop) 
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luation Form 


Course Name (Optional) 
Instructors Date 


Instructions Circle the number that approximates the value you assign to each statement. Poor equals and 
Excellent equals 


Effectiveness of method of presentation 
Instructor@ command of the subject 

The course material was clear and concise 
Length of course was appropriate 

Ample opportunities for questions and discussions 
Course was relevant to my job 

Course gave some good takeaways. 

Course improved my knowledge 

Course will help me improve my performance 
Course met my expectations 

Overall rating of the course 

Overall usefulness of the subject matter covered 
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The strong points of this course: 


The areas in which the course could be improved: 


Other comments you may have that will be useful in evaluating the course and/or instructor: 
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ectiveness Form 


DO- __B Structural Coverage Training Effectiveness uestionnaire 


Complete the multiple choice and True/False questions below and return this form to the Human Resource 
Coordinator. This questionnaire is primarily used to determine the effectiveness of the Group Leader 
Process training you had approximately three months ago. Employees are expected to answer at least 3 out 
of 5 questions correctly. You may use the training material and process documents available on the 
Intranet to complete the questionnaire if needed. 


Name Position 


Date of Training Return Form by 


1. Which of the following is a true statement in regards to structural coverage analysis? 
a. Structural coverage analysis helps to find areas in the software that has not been 
exercised by a set of test cases. 
Structural coverage analysis helps to identify dead code. 
c. Structural coverage analysis helps to find unspecified functionality in the software. 
d. All of the above 


2. True or False: The typical minimum number of test cases to achieve MC/DC coverage is N+1, 
where N is the number of conditions. 
a. True 
b. False 


3. The following statement is a requirement for which DO-178B Level? fEvery point of entrance and 
exit are invoked at least once. 


a. DO-178B Level A 
b. DO-178B Level B 
c. DO-178B Level C 
d. Botha. and b. 

e. All of the above 


4. Which of the following is required to ensure MD/DC Structural Coverage 
a. Every statement in the program has been invoked at least once 
b. Every control statement takes all possible outcomes 
c. Every non-constant condition in a Boolean expression is evaluated to both true and false 
d. All of the above 


5. For which DO-178B Level is MC/DC Structural Coverage required? 
a. DO-178B Level A 
b. DO-178B Level B 
c. DO-178B Level C 
d. DO-178B Level D 
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MANAGEMENT SUPPORT!!! 
» Lead by example | have managers take 
the training 
»Demonstrates priority of effort within 
organization 


»Employees expect managers to 
understand the concepts and practices 
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Sagem Features 


» Establish pool of qualified trainers 


» Identify who needs training and when 
» Pilot projects (e.g. tool roll-out) vs. all staff (e.g. basic 
Statistics) 


» Select best training format for each yPR a q 
» Hands-on training is more effective ~ 
than lecture for tools/processes 


» Coordinate training with tool and proce K \ 
implementation 
» Train immediately prior to use 


AVISTA :: People. Process. Promise. 
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» Capture for later use 
» Video, screen capture, podcast, etc. 


L\ » CAUTION: Dondé go too far 


Seer ) Features 


» Plan for revisions and updates 
» Tools/processes will change -training will need to be updated 
» Refresh employees who have been already trained 
» Update recorded training for new training 


» Review material and effectiveness forms prior to training 
» Independent set(s) of eyes 


» Update/create QA audit checklists in parallel 
» Audit against new processes immediately to reinforce training 


AVISTA :: People. Process. Promise. Page 10 
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People. Process. 


Requestor: Request Date: 

Training Course Name: Projected Training Date: 

Method of Training: Location: 

Instructor: List of Trainees: 

Number of sessions: Number of hours needed per session: 
Description of Training: 


Training Specifics: List any equipment, tools, paperwork, etc that needs to be provided prior to training. 


Required Training Professional Video required? 
Computers If checked, how many? List software needed on computer: 
Projector 
Net meeting? 
Copies of handouts Add to Job function? 
Snag It? 


Training Material Reviewed by: Questionnaire Completed and Reviewed by: 


Detail any other additional needs: 


VP-Operations Signature: 


VP-Engineering Signature. 


uman Resources Training List 


Reserve conference rooms 

Invite Trainees (include the description of the training in the invite) 

Invite Instructor and requestor 

Prepare sign in sheet and evaluation forms 

Notify IT if computers are needed (noted above) 

Notify videographer if professional video is needed 

Notify supervisors of training and who will be attending 

Check to ensure the effectiveness questionnaire is prepared and checked in; if not, notify Instructor and/or requestor that it 
needs to be written 

Set up training tasks for timekeeping 

Post training material (PowerPoint, visual aids, etc) on Intranet after the training 
Enter Evaluation Form data into spreadsheet 

Send Evaluation data to the instructor 

Send out Effectiveness Forms/Questionnaires (three months after training) 


Additional comments: 
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i._|ining Effectiveness 


fEffectiveness data 
Anternal audits le 
‘Supervisor observation 


ICAR’s Training Great! 


Organizational data Effective 


/SCAMPI results 


Act on it 
Retraining is cheaper than failing SCAMPI A 
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=xpa ie Features 
» Effective training system Is critical to successful 
CMMI implementation 


» Training system should be one of the first 
systems implemented 


» Coordination of training timing with new 
tool/process roll-out is very important 
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seed Features 


Contact Info 
Tom Bragg 
Vice-President, Operations 
AVISTA Incorporated 
Phone (608)348-8815 
Fax (608)348-8819 


Email Tom.Bragg@avistainc.com 


AVISTA :: People. Process. Promise. 


Transitioning to the CMMI 
what they never told you 


steve.fried boeing.com, 


This presentation contains information 
that may not be suitable for all 
audiences. 


Audience Discretion is Advised 


AChanging from any one legacy system to 
another can be a high risk, error prone 
undertaking 


AOur planning often overlooks or 
underestimates the “human factor’ when 
undertaking change 


AThe “human factor’ is a big key to successful 
change 


= WIIFM 
= Pain Management 
= Mapping 


A6 Practical Tricks &Tips 
" Tricks they use 
= Tips you can use 


# What's In It For Me? 
> how's it benefit me 
> why should | buy it 
= Must be answered from their 
perspective 
> ego recognition 
+ private or public 
« Natural leader among peers 
> financial rewards 
+ money, gitts, travel 


t 
> Internal standard 
+ It's best for the company 
« pride in workmanship 
= Listening, observing and asking will 
clue you in to each person's WIIFM 
"Sometimes, no matter what, a person 
has no WIIFM 
> management challenge 
+ Special projects, peer pressure 
+» 2X4 


APain Management 
"= Change happens only when the pain of 
maintaining the status quo exceeds the 
pain of change — think root canal 
"Pain, or even a threat of it, is a great 
motivator 
" Tools to regulate the pain levels 
> audits and findings 
> appraisals and benchmarking 
> Strict adherence to Policies and 
Procedures 


= Mapping 
> Is knowing who Is friends with whom 
« lunch buddies, car pool partners, 
“Chatter” buddies, etc. 
> Is tracking who influences whom 
« formally — bosses, auditors, etc. 
+ Informally — SMEs, networks 
" Charts the formal and informal 
relationship network 


T 
=" Knowing relationships gives insight into 
> who can best apply pressure 
> who can informally pass along 
information — both ways 
> who can best persuade 
" Use these relationships to effectively 
use the information network 
" Mapping relationships relies heavily on 
Observation 


= WIIFM 
= Pain Management 
= Mapping 


A6 Practical Tricks &Tips 
= Tricks they use 
= Tips you can use 


Tricks 


Let ‘em Eat erring 


A Red herring that ts! 


AExample: 
"Agenda is about schedule end date 
= Diverter asks why SQA takes so long to 
schedule audits 
"Whole meeting goes off to discuss the 
problems with SQA 
= Everyone got hooked 


ADiversion was successful 


i 


ARefocus discussion when it goes astray 
nterrupt ‘this is interesting but not why 
were here” ...repeat aS necessary 


AYell “Hey, this is interesting but not why 
we’re here” or “Hey, we have a decision 
to make” 


Af all else fails, adjourn the meeting 
AAfterward get with the diverter (and the 
b O 


Tricks 


Gimme the Checklist 


"Project wants to start calculating ROI by 
using their current PCR system to track 
rework 

"Nothing is currently in place, so 
whatever is done is an improvement 
"intent of ROI, its purpose and the 
definition of rework in this context given 
to SW staff 

" Staff explores scenarios, then asks for 
checklist — think AFRs 


= Can not convey intent — just compliance 
"= Generally does not require critical, 
evaluative, thinking 
"Does not provide an environment for the 
exercise of discretion and good judgment 
"ends to translate everything into binary 
AAdopting and embracing the CMMI is not 
translatable into binary 
ADo not get hooked into trying 


AEmphasize what we intend to accomplish 


AGet the users to write Pols and Pros — make 
the process inclusive using Process Working 
Groups, think jury pool 


fexpect that they will have to exercise 
judgment, discretion and thoughtful analysis — 
‘they are college grads, engineers, and we 
pay them well’ — James H 


ARemind them they will live with the results 
(now is the chance to get it done right) 


ou’ve GOT to Love Puppies 


AYou say tomayto 
Al say Tomahto 
Let's call it a “Puppy” and FOCUS on 


the real question — whether or not to 
put it in the salad 


ARise above the semantic noise 


ADon’t focus on immaterial details 


ADeflect semantic issues by switching to 
neutral, “straw man” words — “puppies” and 
“stuff” seem to work well. 


AGet folks focused on the message’s intent, 
what it’s trying to say...NOT how it says it. 


ou Can Understand and Disagree 


AUnderstand = comprehend 
AUnderstand = grasp concept 


AAgree = concur 
AAgree = accept 


Understand <> Agree 


Understand = Agree » CONFUSION 


understand 

ALet people know about your usage up front 
and at the beginning. 

AGet them to do the same...things go faster 
and smoother 

AEnables discussion to focus on either 
resolving disagreement or imparting 
knowledge — and then, maybe, resolving 
disagreement 


A Rose by Any Other Name 


Isn’t a Rose 


Alf it grunts like a pig... 
Alf it smells like a pig... 


Calling it a rose wouldnt lead me to give 
a dozen to my wife! 


AEuphemisms and abstractions in language 
can lead to waste of lots of time 


"Ww h 
=" one company uses “confusion” for 
“conflict” 


AEuphemisms that muddy the “Sender- 
Message — Receiver’ paradigm increase the 
risk and the likelihood of confusion 


Alt’s possible to be tactful, diplomatic and clear 
speaking all at the same time. 


Pavlov Got Guppies to Respond 


ABe generous with rewards 
= public praise 
" gifts — movie tickets, lunch “chit’, etc. 


AAlso, be aware of and sensitive to the 
possibility of unintended consequences 
"Does rewarding your “go to” guy 
> facilitate getting things done 
> glorify level 1 “hero worship” 
"Spring Cleaning at Insurance 


APeople are the ones who make or break 
projects; they are the key to success 


Use the Mindset tools as an overarching 
framework within which to understand their 
motivation and understand how they operate 


Keep the tips, tricks and gottchas in your tool 
kit to use when needed for overcoming 
obstacles, making breakthroughs and moving 
forward 
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Beem Features 


¢ We have problems with poorly written Requirements 


¢ Our requirements are not always testable (I.e., complete, 
feasible, consistent, and not subject to interpretation) 


¢ There is too much re-work in our product development 


¢ We deliver the product when it's completely tested and not 
before and we know what completely really means 


¢ We don’t look for and track defects until system test 


« We always pass from one life-cycle phase to the next clearly 
reviewing and satisfying entry and/or exit criteria 


¢ The product is not used/usable by the customer once delivered 
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Seeger 0) Features 


The ability of a set of inherent characteristics of a 
product, product component, or process to fulfill 


requirements of customers (CMMLDEV 1.2 Glossary) a 


Conformance to Requirements 
Rapid Time to Market 
Low Cost 
Lots of Features 
Easy to Use Useful 
ero Defects 
High Performance 


\ AN SS AN NS SN 


We define quality as many things, but it is up to each 
organization to define what it is and how to achieve it 
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Organization policies, standards, development processes 
and procedures are not followed consistently 
Defects are introduced starting with the requirements but 
not identified and fixed except when found by the customer 
uality improvement is primarily composed of “testing 
quality into the product ” 
Investment in quality is much lower than the cost of poor 
quality 
— Little invested to prevent problems - plenty spent to fix 
Products are delivered that do not match the customer. 
requirements or expectations 
— Poor satisfaction 
— Low confidence 
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Baer) Features 


Product uality Engineering includes all technical and 
management functions that determine the quality policy, 


objectives, responsibilities, and implementation by means such 
as 


— _uality Management - Creating and Implementing Policies, 
Processes, Procedures and Standards 


— _uality Functions -  uality Control and _uality Assurance 
—  uality Planning - Specifying and Measuring — uality 

—  _uality Requirements - Clear, Accurate and Complete 

—  uality Assurance - Process and Product A 
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PPQA Defined 
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= | vs. Quality Assurance 


Product Quality Engineering 


Implemented through 


Quality Control Quality Assurance 


Contributes to Contributes to 


Product Quality Process Quality 


Is defined by Is defined by 


What People Produce 
R ; P How People Do It 
¢ Kequirements 
° Des; ¢ Standards 
cela * Procedures 
¢Code 


Adapted from 
“Software 
Quality 

* Project Level Processes 


Engineering” by 
Michael Deutsch 
& Ron Willis 


¢Test Plans/Cases 
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seen) features 


Quality Control evaluates the Quality Assurance evaluates 


products the process 
¢ Product quality checks ¢ Process quality checks 
— Are the requirements — Is the process being 
testable followed 
—  |s the product within — Are there effective Quality 
tolerance Control activities 
— Will the product meet — Is the process working for 
acceptance criteria the organization 
* ——_uality Control Techniques * —_uality Assurance 
— Reviews of work product Techniques 
(peer review) — Process Reviews 
- Inspections — Work Product Audits 
— Tests (unit, integration, etc.) — Assessments 
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Saeed Features 


Process & Product Quality 
Assurance (PPQA) 


Providing staff and management with 
objective insight into processes used and 
work products created 
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feed Features 


PPQA does: PPQA does not: 


v Reinforce correct x Focus on compliance with 
process Use the model (CMMI)- that@ an 
Vv Objectively appraisal! 
measure process . ' 
compliance x Verify the rgoodnessoof 
7 Gi work products 
pport process 
implementation & x Require the same level of 
improvement auditing on all projects 
v Operate — x Need to be organizationally 
collaboratively independent 
v Provide process 
feed-back 
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Understand the organization and project processes as well 
as the project's needs 


Drive creation of the project's _uality Plan 

— Tomatch project size, risk and criticality factors 
Assist in tailoring the Project's Process so it aligns and 
supports the —_uality Plan 

— Assist in setting up peer reviews for selected work products 


Provide mentoring and other input as to the efficiency of the 
software process as used by project members 


— Discussions with project members and the project leader 
— Analyze the process in the context of which it is being used 
Perform quality audits to ensure quality goals are met 
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ee) Peatures 
¢ Problems ° 
— Management sees 
no added value 


— Expectation to fund 
and resource out of 
current headcount 


— Active subversion 
of a quality process 


ELiCeae[sMeamlUElarcle(sdantc)ilmedel) exeya 


Solutions 


Begin a measurements 
program early — start with 
measuring prevention 
cost vs. cost of poor 
quality (or keep it simple 
and measure cost related 
to rework) 


Ensure adequate 
communications and 
focus 

Provide frequent feedback 


to the management team 
in a variety of forms 
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¢ Problems ° 


— Staffseenovalue — 
inPP A 


— Staffassume 

7 PP Ais 
measuring 
individual/group 
performance — 
— Staff perceive 

PP Aas 

“outsiders” or “the - 

process police” 
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Solutions 


cog nenCeinc 


Improving Software Economics 


select appropriate PP A , 
Staff that are collaborative 
in style and respected 
(knowledgeable across 
the product development 
lifecycle) 

Educate staff(PP A 
orientation) on benefits 
and “what's in it for me” 


Again, choose respected 
PP A representatives 
from within the 
organization 
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Problem 


¢ Difficult ensure that 
PP A activities are 
conducted objectively 
and consistently 


11/16/2007 


©2002- cognence, inc. 


Solutions 


Use pre-defined 
organizational checklist 
and tailor to include the 
project's specific 
processes — but be careful 
of the “checklist mentality” 
(i.e., audit mentality) 


Establish standard 
escalation criteria 


Use standard results 
reporting mechanisms that 
tie results to process and 
not teams and/or people 
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— last but not least! 


Problems Solutions 


» More resourcesneeded 1. What we typically see — create a new 
early on during process functional team to perform PP A 


2. Alternative 1 - Utilize the test team 


adoption | sei 
» Resources (PP A i — and objective PP A 
taffing) a ae , | 
° . Alternative 2 - Establish a flexible 
* Need PP AofPP A cross-organizational “virtual” PP A 
¢ PP A role may change pool - cycle all technical and 
as organization matures management resources through this 


(e.g., measurements) role—coverPP AofPP Areviews 
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= roach Decision 


Things to Think About — no matter which implementation 
you choose 


¢ — Know your organization and current culture 
—  $ize of the organization 
— Geographical dispersion 
— Current knowledge/experience level of the development and test 
—  Qutsourcing initiatives 
— Current level and capability to understand/use defect data 
¢ Empower your PP A Representatives! 
— They must have the “last word” on phase-end deliveries 
¢ — Have sound escalation procedures 


¢ — “Go/No Go” for Exit/Entry Criteria from/to all life-cycle phases based on 
process _ product defects (especially for the “front-end”) 


¢ Adopt a risk based approach to scope extent of PP Awork 
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Lao Function: Possible Benefits 


¢ They understand the what and why of quality functions 


— They see the value added in quality-related activities 
and do not perceive themselves as costly overhead 


— Understanding that a defect, is a defect, is a defect 


¢ They can help to prevent defects from being introduced 
into the system starting with the requirements (if 
included/invited as key reviewers and approvers) 


¢ Can be easily trained or hired with uality Engineering 
training and/or background 
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ey are not part of the development team 
and probably will be viewed as “process 
police” having difficulty in getting cooperation 
from the project team 


They are very often not invited or included in 
planning and key review activities 


May not know the product or stay technically 
involved/knowledgeable, therefore creating an 
‘ivory tower’ mentality or perspective 


Resources chosen for this function are 
sometimes not trained or qualified - “warm 
bodies” 
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They are sate of the development team and are not 
typically viewed as “process police” 


¢ They understand the what and why of quality functions 


— They see the value added in quality-related activities 
and do not perceive themselves as costly overhead 


— Understand that a defect, is a defect, is a defect 


« They can help to prevent defects from being introduced 
into the system starting with the requirements (as key 
reviewers and approvers) 


¢ They are the last line of defense, as such, keenly 
interested that products are delivered to match the 
customer's requirements or expectations 
“rnenor——SSS*~*~<“‘*~*é‘*«OQNENCM ne SsSs*t*~=<CS*~*i‘=~*‘«O AS 


Improving Software Economics 


Feo = ee . 

un ssaem: Possible Benefits - 2 

By ermal in up-front planning activities, they 
can ensure quality plans are in place 


« They can effectively oversee that organization 


policies, standards, processes and procedures are 
followed 


¢ They understand the product, participate in work 
product reviews, and can directly contribute to the 
improvement of work products 


« They can better use process and product defect data 
to enlist management's support of quality engineering 
concepts (now, across the lifecycle, not just test) 
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¢ Continued resource over-allocation with additional 
responsibilities and training needs 


¢ May further the divide (if there is one) between test 
and development 


¢ Qutsourcing 
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es sal: Possible Benefits 

“a’ development team (i.e., “one of 
us’) and are not typically viewed as “process police’, 

even if they are from a different product/application 

group 

They can help to prevent defects from being 

introduced into the system starting with the 


requirements (as key reviewers) 


They understand the product, participate in work 
product reviews, and can directly contribute to the 
improvement of work products 


As team members/managers are cycled through this 


role, they begin to understand the negative or positive 
effects the process may have 
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eee Features 


Proactively scheduling their PP A work — so it 
does not get done in the “margins” 


¢ Difficult to keep a virtual team 
— Trained 
— Communicating effectively 
— Focused on the same objective 


— Performing consistently across the team 
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Overlap — “.....with one stone” 


emeed Features 


Leveraging existing skills and activities 


PP A Test Virtual 


iq) 


Planning for quality 

Designing in quality factors (e.g.,maintainability, reliability) 
Establishing the use of standards and procedures 
Reviews 

Testing (Dev only unit test) 

Audits 

Setting quality goals 


Providing visibility into the process and product quality for 
management (Reporting) 
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Ensuring non-compliance issues are resolved before the 
product is delivered to the customer 
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When planning the project, use a pre-determined risk 
based PP A Planning Checklist to decide level of detail 
and frequency of PP A oversight and involvement for 
each project. Possible considerations 
a Project size 
Complexity 
Project Duration 
Business risk 
Technology risk 

uality Objectives/Requirements 


This can appropriate PP A resources where most needed 
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Customize PP A review and audit approach to fit different 
project needs 
— Artifact Audit 
Ease of scheduling, minimizes overhead _ subjectivity 
— Reduced opportunity for discussion, questions/answers 
—  uestionnaires 
Minimizes overhead 


— Lowresponse rate, permits interpretation issues, reliability 
is questionable 


— Interviews 
Enables discussion, questions/answers and clarifications 
— Difficult to schedule meetings, can be subjective 
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o Project Management Processes Prrojyect Warragement Processes 


Toll Gate Checkpoints for Approval 


pproval/Funding for Analysis and Design 
effort 


Key Project Management Product Deliverables 


Estimates 


2. Detailed requirements approval (RR) 
Architecture Review approval 


. High-level project schedule & budget estimate (planning) 1. Planning estimate - high-level 


3. Approval/Funding for Construction to begin 


. Detailed requirements, Architecture Review 2. Commitment estimate - detailed 


1 
2 
3. Detailed design, project schedule, budget (commitment) 
4 


CDR 
3. Revised cost estimates - ( ) 
— ; throughout project life cycle 4. Change request approval as project phases 
. Product build, test, training and contingency plans, project progress 


control documents. 


if PPQA Reviews 


. Post-implementation review reports 5. System Testing sign offs (TRR) 


6. Client approval for implementation, Risk and 
Change Management approval 
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uality Control ( A testers) evaluates the product 
uality Assurance (PP A) evaluates the process 
Marriage of these two provides full cover of product 
and process creating one robust function — “The 
whole is greater than the sum of the parts” 
Leverage resources and skill set of the team to review 
process and project artifacts 
Facilitate better quality planning activities 
Use risk-based approach for PP A planning activities 
and resources on projects 


Ensure product quality at each development phase 
and before the customer sees it 
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» Lessons Learned 
» Planning 


» Measurement 
» Process Implementation 
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» Other Tips 
» Summary 
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» Safety-Critical Software Engineering Services Provider; 
Specialization in: 
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with programG execution schedule 
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Compliance reported at Monthly Program Reviews 


Systemic issues reported at quarterly Quality System 
Management Reviews (QSMR) with division staff 
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ocess Structure 


Integrated Process Manual (IPM) 


Program 
Management 
Processes 


AProgram Planning 
AEstimation 
PN TroNele- iM edaliclalare Melt Mexeyilixe)| 
ASupplier Acquisition 
WM ETareYe(cVaat-vals 
AChange Management 


Program 
Life-Cycle 
Processes 


ARequirements Analysis 
ASystem Architecting Design 
ADesign 

ACode and Unit Test 
AFabrication and Assembly 
AProduct Integration 

A erification 

A alidation 

AProduction 

AField Support 


Program 
Support 
Processes 


ARequirements Management 
ARisk Management 
AConfiguration and Data 
WM ETareYe(cVaat-vals 
AProgram Metrics 
ADecision Analysis and 
Resolution 
AWork Product Inspections 
ADesign Review 
A uality Assurance 
Alntegrated Moke [Kj (ex-Mee le] o) oLolat 


Organizational 
Processes 


AProcess Improvement 
ATraining 
ADivision Metrics 
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In keeping with the Antegrateddaspect of CMMI®, the role of 
Quality Assurance has been redefined to encompass all of 
these integrated processes 


Quality engineers are moving away from end-line inspection 


relalemrelulel|t 
| More of an upfront, cross-functional, and consultative role 


Not strictly manufacturing- and product-based 
| More business- and process-based 


(YAW 0) co)vi(0 [otsmr- a= (ele\Vcmrel a(@MVAK)1e)(om [ale ly elo) ae(csal meral-1e).@r-lale 
balance 
1 Ensure products and services are of the highest quality and 
create customer value 
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e 


Integrated view has blurred the distinction between 
HW QEs and SW QEs allowing for more consistency 
across programs 


NWitelcemclarcle)(srem (om el-manlaer-lalyacvile)arcl mle [(s) aecme) | 
orarelale [ave 


(©) Stow alcrolo Mi Com at-\\iomr- erelan) ©) ccvalclarciiomUlale(cvacir-valeliale 
of all facets of the business rather than a limited 
Ulalolclesirelaleliale me) me)alom- Kore! 
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QE is not expected to be an expert in every area but they are 
expected to help focus on effective and repeatable process 
le) geleli (eres) 


QEs conduct Integrated Process audits that span all of the IPM 
processes ie) more lela 6) cele] eelae 
DY <i) dan) alsxom ©) cele] e- lan 6) celercssyomere)en)e)it-lalecom Cem talom | ealy 
Process Compliance Monitor (PCM) tool used as the audit 
checklist 
Compliance scores are entered into the PCM tool as the audit is 
performed 
Process compliance automatically reported monthly to 
aarevarelesiparciaye 
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the teams they support is no longer based on the old 
view of the rfpolice forceo 


BYoliXclana Mola lalsverd(e)areleloiavicl-la miele elce)(s\eimr-lalemuals 

Quality Engineer 

lun =)a=y-1,4/ale mele) me) mual=ms)\e)V{-10)| el-manlsvaltcUliny 

| Stronger, more effective customer relationships (both 
internal and external) 


Desire to be a fteam player, Oto fit in,oto mot cause 
any wavesocannot take precedence over fMdoing the 
ale laimialiarele 
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Program is not surprised by QA audits 
| Processes to be audited are well known 
| Expected evidence from performing the process is 
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Overview 

A brief description of the process intent 

Entry Criteria Exit Criteria 

State, Prerequisites, Criteria State, Criteria 

Inputs Outputs GCSD INTEGRATED 
Needed work products, resources Resulting work products | PROCESS MANUAL 


Required Activities | | $-002-001 
Mandatory tasks to implement the process 


Measures 


Process performance against plans 


Organizational Improvement Information 
Metrics, reusable work products 


erification 
Process compliance oversight 


=a ; = fee Program evidence needed | 
Tailoring Guidance |_| to demonstrate IPM 
Approved tailoring, process specific process compliance 
Implementation Guidance 
Common implementation descriptions 
Supporting Documentation and Assets 
Applicable organizational references 
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SE-Titedatare 


Program Plans pragram 


Program process baseline Start-up 
maceleleclanm=y.<cleeliele SrearamiPhase 
Compliance artifacts meen 
QA verification ____ Program Appraisals | 
Nfolatrexe)an)e)ir-valetomaniiter-liie)e 
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2 Organization 


Wikelarele(svaatcialmrslmrellmcaV.co ism ale) im arotsmlarsye| aimee 
process compliance as determined by QA at any 
gals 

a OF \/ Mn ele) 0) c0)V/(0(~xomrer-\ 6r-10)|/IAVancomce)|mUlOmialcomel-lromrelt 
multiple levels 1 program, product line, business unit, 
group, division 

Systemic compliance issues are easily identified 
across multiple levels 

OyANW/Felarelo [eva atc)aimn 0) ce)ale(-xour- Waele) alual \vanaos ele) eure lare, 
analysis of the compliance issues and top process 
issues 
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pliance view 


Status FiteringExport Process 
ARepresents overall process — [_Bcess 


compliance score for program 
ABased on lowest color score i harsh, {EXAMPLE PROJECT} 
but in keeping with CMMI standards Baseline Revision: 93 IPM Version: IPM v6 


| NI Project Score 


: : Featen : ; ee Scoring 
ADepicts scoring distribution over all a yy Distribution 


process items 


A : x . Ni: 7 PIO LI: 16 Fl: 24 NS: 485 NY: 0 NA: 24 Rescore: 503 Total: 547 
More insight on overall project score 


ADepicts score for each process 
executed or being executed by this 
program 

A3 columns identify types of processes 


| Export Project Status 
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Overall 


Integrated Logistics Support 


Requirements Management 
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More focused on integrated processes and 


(e) Colerexosom ore) an] @)if-lalers 


mac lecelle QAM icmsialiitiaremiceanme mm cct- (ethics 
detection" model to a "proactive predictive" 
aateye(=) 


Teams are leveraging QA and using them to 
help improve their processes 
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| Customers (internal and external) still depend on and 
expect QA to have the courage and integrity to make 
ialemare lao me (sterci(e)atomiarolmcye] ©) ele)amnc (e)ialemialomalelaimualiarele) 

in OYA c=) F-Ulatsmlesme)aer-lalyZsltle)arelmlalels) el=)ale(>)alesmcom-li(e)i) 
vaTsvagincom (elcalalavamalelarcexe)pale)it-lprerctom-lalemacrere)aalpalciare! 
corrective actions 

Overall the redefined role has had a positive impact 

for the Quality Engineers, our programs, and the 
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Office of the Chief Information Officer/G6 


U.S. Army Aviation and Missile Command 
(AMCOM) 
Redstone Arsenal, Alabama 


Lean, Six Sigma and CMMI... 
Working Together to Achieve High 
Success 


by: 
Susan Bassham 
November 2007 


Office of the Chief Information Officer/G6 
Why Lean, Six Sigma and CMMI? 


¢ AMC/DA directives 
¢ Supports Software Engineering Process (SEP) 
Improvement 
¢ Integrate and compliment one another 
¢ Gives insight into reality 
— Data-based decision making 
— Measure variation 
— Know when to react 
¢ Used for problem solving 
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Office of the Chief Information Officer/G6 


Lean, Six Sigma and CMMI 


All are process improvement methodologies 
Lean focuses on speed, efficiency, and removing waste 
Six Sigma is aligned with precision and accuracy 


CMMI supports best practices for software development 
including risk reduction, increasing efficiency, and 
improving the overall quality of products and 
deliverables 
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Office of the Chief Information Officer/G6 


Organization Background 


Small, Federal organization (140 civilian/contractor) 
Achieved SW-CMM Level 2, August 2002 
Lean approach implemented, Feb 2005 


Achieved CMMI Level 2 and Capability Level 3 in 9 
Process Areas rated, May 2007 


Working toward CMMI Level 3, Class A appraisal 
planned for Mar 2008 
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Office of the Chief Information Officer/G6 


Lean Improvement Benefits 


VSM Conducted Feb 2005 resulting in 14 Kaizen 
events and many tasks 


Kaizen improvements yielded new Software 
Engineering Process (SEP) iterations 


Improved metrics — data driven management 
Eliminated unnecessary steps identified as wastes 
Provided customer’s first critical needs faster 
Test driven approach focused on FTQ 

Improved workload/workflow management/staffing 
Better communication through teaming 
Reduced backload 


Office of the Chief Information Officer/G6 
SEP Processes Rolled Out 


¢ The SEP processes have been rolled out incrementally by 
versions since June 2005. SEP Implementations were: 
— June 2005, Version 5 (Interim) 
— December 2005, Version 6 (Loop 1 Rollout) 
— February 2006, Version 6.1 
— February 2006, Version 6.2 
— March 2006, Version 7 (Loop 2 Rollout) 
— July 2006, Version 8 
— August 2006, Version 9 
— September 2006, Version 10 (Loop 3 Rollout) 
— November 2006, Version 10.1 
— January 2007, Version 11.0 
— February 2007, Version 11.1 
— April 2007, Version 11.2 
— August 2007, Version 11.3 


— October 2007, Version 12.0 (CMMI ML3 Remaining Process 
Rollout) 
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Category 


Metric 


Current State Baseline 


Future State 
Projected 


Office of the Chief Information Officer/G6 
VSM Projected/Actual Results 


Total # Jobs/yr 


Tot # AKO pgs 


25 


> 50 


Backlog 


21 CRs 


< 10 


Lead Time 


374 days 


< 90 days 


Changed goal 
< 105 days 


165 


138 (G6 Ready for 


Production) 


Process 


Phases-Loops 


9 


4 


4 


# Activities 


63 


24 


22 


# Approvals 


65 


18 


ri 


# Documents 


25 


11 


8 


2(CMMl) 
CL3 in 9 Process 
Areas 


CMM/CMMI Level 2 (CMM) 3 (CMMI) 2 (CMM) 


Process Maturity 


100% FISMA, AR 
70-1, etc* 


0 
DA Compliance 100% FISMA, AR 


70-1, etc* 
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Office of the Chief Information Officer/G6 
FY 08 Projected Events 


SCAMPI-C (November) 
SCAMPI-B (January) 
SCAMPI-A (March) 


Value Stream Maps/ 
SEP Readiness 1 — VSM (May) 
Reviews 


1 - Workplace Organization 
Kaizen Events 5 — Projected from VSM 


Jan — SCAMPIB Results 
z Mar — SCAMPIA Results 
Town Hall Meetings 34 Qtr — SEP Version 12.2 Rollout 


1 — Risk Management 
1 — Functionality Test Cases 
1 — CMMI v1.2 training 
Training 4 — SEP Version release training (V12.1, V12.2, V12.3, 
V12.4) 


Office of the Chief Information Officer/G 


Value Stream Map 


CIC Software Engineering Process (SEP) 
Future State Map 
31 Jan — 3 Feb 2005 Note: Does not include 
closeout Loop 


Design and Devel 


Implementation Phase 


Supervisors 
‘ing 
Group 


Planning Phase 


Scheduling 
Board 


Loop 1 
P SPL 
5 | 
| Design’ | Time | |- Flow charts - Screens | STDs | Functional | Mave app to 
| Create estimate |. Screen - DB |. 508 | Security Production 
acceptance | Labor hours layouts - Reports | Documen- | Readiness 
jtest cases estimate | |- Report Itation 
| Test |- Cost SoM; SPL; layout SPL; ca Cust, SP} Dev SPL; Dey, PU D 
| Peer review| | &"o" estimate bisa a ta Dew’ Sa - Interfaces be Peers bev Dev Mgmt Bey, Peer Rev, Dev DoM Cus, Dev oom 
|- Architecture Sa4 Dev Senna. | Tested SH Dos pe Pl eeick 
| Risk Rep | Testable Ls Ree Soa 
analysis Documented : 
|. PSDT | Approved Fielded App 
| | Traceable SCRE 
SPL | "Right" “Teste 
& & amount d Te! Detects Eom 
Cust prove: Complete = q raged, Wecllozsh 
ua pet Bae Approved emai | Se DB Design DB complete OK captured; ie Aid aes ae ese FAST isa aaa 
Pl 9 Plan oY App Product || |rework; to std eH (compliant; App Data accepted - Tracked! 
= 508 comp ip TEM) App ready Measures 
Fi fin Caneel for fieldin = Closed 
| Design eae 
ine 
D Dat 
Rad | | 
Pie! Doo 
Spec 
On-line 
Help 
5d th 2d 1d 2d td 2d a 2h 2d 1d th 1d 1d 2d 1d 1d 1d ad 1d 
30m 4h h 2d 2d ad 2h 3d 2h 6d 3d 1d 4h id 16d 2h 2h 1d 4h 5h 1d 1d 4h 
TLT = 
< 
TCT = 
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Loop 0 


Establish 
Product 


CIO-G6 Software Engineering Process (SEP) 


Loop 0: Concept Definition & Product Requirements 


Goal is 30 days 
31 October 2007 


Decision Tree WG RE } : 8. Total Cost Estimat — — Overarching 

Staff —lnitiate or Update *Previe e 7 Sa = as Activities 
eSuspend ap 
eFeasibility Study CM QA Security Metrics 
More Customer Info Plan | J Plan | V Bian Pen 
eReject 

oT 

Plan 


; Draft || Final ; 
le Finish product reqts o ; RMP ‘SPP QACR | |CMAFRS| Risk Management Process: 
le Negotiate jobjs? size 4 ts 
EEE le Determine le Negotiate timeframe Exit Criterias le Identifying risks 
chedule meeting fExitCntenas——] Lifecycle le Analyze/plan risks gmitigation’ Functional reqts by sisi fae 

je Conduct meeting le Product set up in DOORS |e Prepare Top Level le Link mitigation to requirement component Revise and Update Exit Criterias je Analyze/plan risks (mitigation) 
le Capture data &PVCS Design sbaselina {create new reqts if necessary’ © Job reats SPP s Testable, validated + Link mitigation to requirement (create new 
le Set schedule for customer meetings) |» \iotification to PT and (MitLoss.o Reamarrare le Request timeframe commitment ® Interface reqts CM audit baseline Cls eipinenest requirement) 
le Set expectations SLIM Team of product Checklist E Es i el a - for PT Performance reqts Review overarching Tech approach le Peer review — determine/review probability, 
| Get customer commitments name and ID le Identify Interfacefsz]|® Top Level Desigi le If known’ Request additional © User documentation plans {CMP, QAP, © Preliminary design loss, and exposure for risks (metrics) 
e Identify risks fe Voice of customer & Enterprise le Erototype fifneededk | | development stalf reats Sr sti Mester © Constraints’ Verify requirementsirisks 
le Collect requirements and record in ®Scope/ description || Integration in eisisks le Review Customer Input Q&A and | |#Nor-functional reats alignment and Seu lare eT digsr malls 

DOORS Problem statement || DOORS SPP compare to DAR-Capability Matrix | | ® Scope adherence abelve abies aciG atta oe sa udleorarol isits/a 
le Initiate Software Product Plan Ounchicn hamid to-determine programming option #User performance reqts | Validate requirements |] | Mincioneccnenlie requirements 

SPP} © lob expectations ’ le Determine job integration © User training reats with Customer e Addl eect le As newrisks and/or requirements are identified, 
Je Clarify requirements within PT # Key date/delivery elements and sequencetn *Secuntyiprivacy regis |p Initiate Job Plan Siccfiverelpmaicnelan follow change management process 
Je Ensure PT records job level effort in drivers ‘ BOORSbb Plan * Accessibility reqts ® Audit Findings Report l* Report/communicate risk scores as necessary 

Date Collacon'Spreadsheet © Customer references, Initiate Information Sharing * Installation reqts © Select "Move to Loop when and if elevated 
le Gather additional requirements as acronyms, definitions, Agreement and/or MOA, if necess ® Special SW reqts 4" in DOORS 

needed for Portal Products and abbreviations |° Baseline functional requirements ® Maintenance reqts Functional Reats. 

© Customer module in DOORS » Productjobjst size, Module to tigger end 
responsibilities Je Peer review timeframe b of Loop CM and QA 
’ \ e Risks Fix reqts @ Define Risk and Mitigation ka activities. 
! © Product Team effort ~Pass reats jo Programming eption (£@®) || © completed Loop 0 
a data -Determinefreview Pgl rating for decision = Checklist 
risks Je\ | (* Defined, planned, analyzed, ~ 
-Verify reqts wy igs and mitigated risks/reats 
Aseee sche <= | |» Verified Functional Reqmts 


SPMGT-LP0O-FDGM 
31 October 2007 
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Office of the Chief Information Officer/G6 


Loop 1 


wwreammTe begend CIO-G6 Software Engineering Process (SEP) 
For output Rots . Loop 1: Job Planning & Analysis 


Goal is 15 days 
31 October 2007 


CDAS i 
Production per 
Scoreboard 


Overarching ee eee eee ey 
Activities 
cm | | aa Rees Metrics 
Plan Plan Plan Plan 
OT 
Plan Risk Management Proce: \ 
e Identifying risks i 
@ Analyze/plan risks gmitigation? 7 
@ Link mitigation to requirement gcreate new 
requirement? 
=» Peer review — determine/review probability, loss, and 
exposure for risks imetricst 
le Verify requirements/risks 
@ Track, manage, and control risks as requirements 
@ As new risks and/or requirements are identified, follow 
change management process 
* Report/communicate risk scores as necessary when 
a and if elevated 
_ Seen 77 Signed|| — 0 |pepaen 
Reviewed RMP—|_='S°—> | Final ||Updatedll Baseline set in i | QACR ||CMAFRS| 
® Analyze & draft preliminary | Ifno changes from | ni Up "| DOORS IAW | 
le Conduct PDR PDR, then ls Rebasclinetae RMP || SPP {I checklist 1 
le Identify Interfaces proceed to Activity9 fupehesal ~ —! © Obtain customer — Exit Criteria: 
® Assign Requirements le Obtain MOAVInfo Sharing Foghat I signature le Approved RMP 
Complexity using DAR Exit Criterias Agreement itailorablet | Pedde cD S fl © Software Product |e CM Audit 
Matrix i AWeohacklisttit Milestone Review | cingi 
* Review criteria in checklist & |, eR LER if Preliminary Desian Criteria | Exit Criterias | © Audit baseline Cls ese Rept 
determine altemate Custom Product? le Electronic Document | inthe PBR? le Updated RMP ] le Select "Move to 
approaches * Programming Decision @ Screen mockups include | le Revise, finalize, I Loop 2° in 
® Determine job integration le Preliminary design main menus | baseline Preliminary Doors 
elements and sequence in |, Constraints/assumptions le Graphical illustrations will Design ! Functional 
Job Plan © Deliverables be numbered sequence - | je Revise & finalize i} Reqts Module 
® Generate SLIM Job Estimate}, Add'l resources/cost include description I requirements and I to trigger end 
& Control Files creraleoaiontinaee - include interfaces | vob Plan op of Loop CM and 
Effort © SLIM job estimate and le Could be edited screens 4 je Update t \ | QA Activities 
-Schedule aaniraliflt for existing products (Exit @riterias SPP \ I » Completed 
® Submit Dev QSADS, if le General work flow |p Reviewed ‘7 | Loop 1 
necessary for new database preliminary design & - naa Checklist 
or server space schedule 
© Basolnetay 4 ‘\ @ Meeting Minutes. 
le MOA/Info Sharing 
Agreement 
ftailorablet 


JMGT-LP1-FDGM 
31 October 2007 


* TL role activities support product level view. 
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WW Team MTG 


sP 
CM QA P 
Plan Plan Seeutty 

Plan 


Loop 2 


CIO-G6 Software Engineering Process (SEP) 
gn & Development Process 


Goal is 30 days 


31 October 2007 


Risk Management Process: \ 
Identifying Risks it zi 


Analyze/plan risks (mitigation » 

Link mitigation to requirementt (create 
new requirement) 

Peer review — determine/review 
probability, loss, and exposure for 
risks (metrics) 

Verify requirements/risks 

Track, manage, and control risks as 
Tequirements 

As new risks and/or requirements are 
identified, follow change management 
process 

Report/communicate risk scores as 
necessary when and if elevated 


OT 
Plan 12. *Critical Desig 
Review (CDR)/IPR QACR |/CMAFRS 
a A - Plan agenda }- Assign work |- Test (duri |- Schedule meetin i il 
11. *Detailed Design F jonality |" Test (during ‘9 - Provide scripts to 
ig) |- Send preview of - Include functionality development) - Informal desk-side, as Bonar eee 
documents to participants test cases ~ Unit test (ref. needed - Mave code and 
- — - Expand preliminary - Review Design = Send sat standards) }- Conduct per schedule compile Exit Criteria: Staging 
Discussion: =» design Screens assignment to - Conduct |- Prepare and send out - Identify version _- Database and 
RMP th \ - Define screen report Reports LAD and Devs) functionality agenda => number compiled code 
Schedule | layouts tess. -Review and clarify test cases - Provide status ¥ - Audit product and scripts in 
Preliminary \ wwe - Interfaces (data, Exit Criteria: - Address changes, as functionality test cases | _ Conduct informal ~Resources SR baseline staging 
design ee Neu beve, frequency, method, |- Verified design necessary (update if needed) peer review of code - Hardware - Audit Report 
ule know outside of wagon || format) - Ensure Cost Write code following | _ Baseline source code ~ Software - Select “Move to Loop 3” 
Send out agenda | wheelie.addiional_||_ Workflow/fiow charts | Development - Schedule Surrant standards IAW checklist ~ Product/Job in Doors Functional 
Identify participants | __develeners)_|/_ Design database Environment is set | |- Resources comment ode - Ensure that work - Demo's Reais Module to trigger 
; ieee’ - ERD diagram (for ue Heldwate - Address change, as assignments are - Software Product end of Loop CM and QA 
Required participants: products containing a |- RTM (DOORS) Software necessary complete Milestone Review Activities 
SPL database) - Baselined Personnel - Check in code to PVCS) _ Ba coline product - Address change, as |- Completed loop 2 
LAD Exit Criteria: - Flow diagram functionality test }- Begin building online necessary ‘checklist 
Developers Clarification  \Wrte functionality cases AW. Required participants: |Exit Criteria: help \ 
Work teat aagee |_chacklist - PT - Customer - Request staging f Required Participants: 
peal ilies Sse (obtain input from —_|- ERD if you have a | |- Customer SO cee Hen Deck Ticket sat’ “Customer Exit Criteria: 
EP i database . 2 
TL LAD, If necessary) LGacennaea Invite as optional: meeting minutes | |- QE/Tester conducts eonducel ke 
Product Indep. Testers layout - Dev(s) |- Minutes/notes informal partial FAST Optional - Minutes/notes 
CMA - Updated with LAD/Dev during Attendance: [> Customer 
functionality test development - Dev(s) approval or 
cases, if concurrence 
z Exit Criteria: Write Code & Test 
‘on progress 
eae i: |- Completed and tested code (eal 
fers el - Updated RTM resting 
peice i }- Verify staging environment allocated minutes) 
i j rene cit ign |- Work assignment - Updated 
eck |- Close out taskers when complete design, if 
PEST - Create database, if needed ‘ 
i necessar 
Activities 11 and 12 EN? be - Notification to SPL code is ready for 2 


combined and performed 
during PDR in Loop 1, if CDR 
criteria is met prior to PDR. 


Functionality Test 
- Document test results in the functionality 
test cases module in DOORS 
|- Provide feedback to LAD on partial test 
|- Baselined source code IAW checklist 


* TL role activities support product level view. 


y, and 


JMGT-LP2-FDGM 
31 October 2007 
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ffice of the Chief Information Officer/G 


Loop 3 


CIO G6 Software Engineering Process (SEP) 


Loop 3: Test and Implementation 


Goal is 30 days 
31 October 2007 


eats 


9. Production 
19. Production —_| 


8. Customer Accept 
Pesenionencesecatgeut Deed 


WW Team MTG 


5. Schedule/Milestones 
4. Completed Reqts 


jpeeeetalinplom ee 


6. Development 


5. Risk Management 3. Defects 


5. Planning & Analysis 
ee Snel 
4. Requirements 


4. Functionality Test Cases | 2. Actual Hours 


Risk Management Process: 


= |4. Requirements _| 
« — Identifying Risks ©. Data 3. Non-Functional Reqmt: 1. Estimate 3. Staffed 
« — Analyze/plan risks (mitigation) L Calls = fF 
¢ Link mitigation to requirement (create new 2. Functional Reqmts 2a CReview 
requirement) 1. Preview 


ae h 
¢ Peer review — determine/review probability, loss, Customer Input 


and exposure for risks (metrics) 
e Verify requirements/risks 


Final Dataset on Job 
- Hours and Cost 


* Track, manage, and control risks as requirements - Code Storage 
¢ As newrisks and/or requirements are identified, - Scripts 
CM QA follow change management process - Completed Customer Survey ! 
Plan Plan « Report/communicate risk scores as necessary ie LT 2. Audit Schedule 
when and if elevated - Signed ASEC 4 Produet 

- Defects by Loop/ Activity 
- OTD 
- Cust. Product Acceptance 
- Customer Job Acceptance 
|- # Trained 


|- Compliance Data — CM, QA, IA 
- Lessons Learned 
|- Testimonials/ Kudos 


7 QACR | | CMAFRS 

e Product Demo ¢ Provide Functionality Notify QE/Tester that Schedule training Move Code Cust. Acceptance (in 

e Ensure test readiness Test Cases to product is ready for Conduct Training Re-compile SYN/Chg) —___ 

Required participants: Customer testing Exit Criteria Complete Prod Publish Final User Store. Code SMe 

- SPL © Train Testers (Cust.) |Exit Criteria ¢ Conduct functionality Ready system QSADS & submit /Exit criteria Doc ¢ Update data(In SPP) Je Satisfied 

- LAD if necessary e Customer Tested & test on job Compiled code help deskticket |, Users © Notify WW that . Collect, Analyze, Customer who 

- Dev e Record & Rework Staged Product Document defects Completed Prepare ASEC Trained & product moved to Report: accepted job, 

- Customer ——— defects; Retest ready for Training Send defects to online help Obtain ATOATO| — Feedback in production ~ Customer reported . 

Invite as optional: Exit Criteria e Conduct Useability independent testing developers for rework Functionality Create Production| Info. System Exit Criteria Satisfaction satisfaction to give 

- TL es Lest Testing (Optional) je Acceptin e Run AccVerify or Tests pass! Baseline e ASEC e Available ~LT CDAS a 

- QE/Tester Readiness SYNERGY/Change JAWS as needed Verify * ATOMATO Working -- Quality Completed, 

- CM/IA le Updated }° Audit Findings. e Run production configuration E ipraduck Approved - OTD Measured, 
Beslan, tf D Report fielding test items Baseline Product la roduc Reported, 
necessary A@m\ |* Tests Run security test * Completed e Customer Satisfaction Auditable Job 

, - Functionality Sondier eenduct FAST Survey ¢ Completed Loop 3 
be Test Cases integration testing Test readiness e End Job checklist 
|» Security - Integrate Baselined 
|-Integration Interfaces 
- Test Interfaces 
ee 
JAW checklist 
© Auditproduct 
baseline - 
© Start ASEC c} 
= 


11/15/2007 


Update SLIM Control Weekly Throughout Loop Customer Satisfaction 
# Exceptions Delivered oo - (Process/Product) JMGT-LP3-FDGM 
* TL role activities support product Pesdinime # Process Deviations | 34 October 2007 
level view. # Product Defects 
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Office of the Chief Information Officer/G6 


Lean CMMI Benefits 


Repeatable, standard process for the organization 
Reduction of defects through Peer Reviews 
Improved planning activities 

Improved testing activities 

Better management insight into products 


Improved communication through Process 
Configuration Control Board — Peer Groups 


Adherence to process with improved PPQA process 
Improved metrics — data driven decision making 


Target Capability Profile 


5 
Rated 
Target -.—-— 
_ 4 
® 
o 
_ 3 
> 
= 
B2}-—--—-—----—---—- 
© 
o 
oh | eee a 
0 
a 


Engineering Project Mat. Engr. Support 
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Office of the Chief Information Officer/G6 


Improvement 


Office of the Chief Information Officer/G6 


Six Sigma Benefits 


¢ Improved measurement process 

¢ Manage and make decisions by the data 

¢ Controls in place to maintain improvements 
¢ Know when to react 
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Office of the Chief Information Officer/G6 


Six Sigma Black Belt Project 


¢ Goals 


Reduce Loop Lead Time by 10% 


Reduce Lead Time for Loops 0-1 from 113 days to less than 
100 


Increase meeting Customer Negotiated Delivery Date by 
10% 

¢ Increase number of jobs delivered by 3 during the FY 
¢ Projected Cost Savings 

¢ Job average calculated ~ $51,484 


¢ Increase productivity by 3 jobs each year ; \eSe" 
¢ Savings of $154,451 annually tt 
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Office of the Chief Information Officer/G6 
Average Lead Time for FY07 Jobs 


Lead Time(Days) 


@ Customer Acceptance* 


O Testing 


© Development 


@ Planning** 


H Requirements* 2nd Qtr Average 3rd Qtr Average 4th Qtr Average 


Quarter Averages 


* Requirements and Customer Acceptance states rely heavily upon customer involvement 
** Began capturing Planning Lead Time in April 2007 
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Office of the Chief Information Officer/G6 


DMAIC Methodology 


Define | (-) |Measure |--)| Analyze] --|Improve] > | Control | 


Structured, data-based, problem-solving process 
— Specific activities in specific sequence 
— Gathering data to facilitate decision making 


— Using problem solving techniques to ensure elimination of 
the problem 


11/15/2007 


[Define | -->) [Measire|— |Analvze) improve] > (Gentroll 


11/15/2007 


Develop the charter 
Review existing data related to problem 
Draft a high level process map 


Create plan and guidelines (scope, how to measure 
success) 


UDefine|) ) [Measure |) |Analize] —|impreve] > |Gentrol) 


11/15/2007 


Heart of what makes LSS work 
Evaluate existing measurement system 
Observe the process 

Gather data 

Map the process in more depth 


a 


11/15/2007 


Use data to confirm source of delays, waste, and 
poor quality 


Stick to the data 

Look for patterns 

Find root causes 

Identify the most critical factors to control 


EDefine!| > [Measiire|— |Analyze} —>|Improve] > (Gentroll 


11/15/2007 


Make changes that eliminate waste, reduce costs, 
and are tied to goals 


Identify a range of possible solutions 
Review Best Practices 

Develop criteria for selecting solutions 
Pilot or simulate solution 

Plan for implementation 


Ebefine!| —) [Measure] > Analyze) —»|iimprevel > [Control] 


11/15/2007 


Purpose: make sure the changes will last 
Document new, improved process 
Conduct training to the workforce 
Implement the change 


Office of the Chief Information Officer/G6 


Summary 


¢ Lean, Six Sigma, and CMMI work together for success 
— Improved Quality 
— Data based decision making 
— Maintain improvements 
— Repeatable process that is standard across organization 
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Office of the Chief Information Officer/G6 


Questions? 
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LOCKHEED wantin} 


Leveraging Lean Six Sigma and CMMI to 
Improve Program Performance 


Elizabeth A. (Beth) Clark 
Mary Lynn Penn 


Moyes -¢al=-em iu reladia 


rurpose 


A Within Lockheed Martin, Lean Six Sigma and 
071i: | o) o] or- Tet al-\-mcomerolaliialerelet-minal e)aed{-Vial-vals 
eT veMm ole alae Mlaltcye|e-ltctemcom-Lolelc-t-t-m tal- me [ey-| Mey i 
ol=)accYou mo) weolele-lanm ol-]uce)aait- lave: 


A Identify where LM business units have used 
Lean Six Sigma and CMMI ina complimentary 
narclalat=) mm Kom iin) o)aen (=m ol cole | e-laam-lalem olerylalst-y 
performance 


A Offer ideas how your business can integrate 
Lean Six Sigma and CMMI 


Vetining the ferms 


A CMMI_- Framework for managing processes and 
integrating activities across an organization 


A Lean - Set of principles for efficient and 
effective processes, based on the Toyota 
Production System 


PNG Pe Motes eat-VsXe Ml F-Vatiamow-le) o)cer-lela cele 
vooyanlelialiace m M=x-lam-lileme)> @=)(¢] uit: Mmmm =r: 1a) 
Processes Operating at Six Sigma Capability” 

A Six Sigma - Problem-solving approach that 
addresses specific improvement needs through 
improvement projects and statistical methods 


a 


OFeyaelanrelameaar-lalerks)at-j4 (es 


A Focus on eliminating defects and rework 

A Reliance on measurement and statistical 
methods 

ENS ral elareiemeyaM late (stesieclacellareMelalemactelereriare 
variability 


= 


Vitterences Create Opportunities 
Lean Six Sigma Cumre 
AAssumes processes have been A Focus on defining management and 
Tel=JalijitexeMelalemel-vilalexe| i=Yoq al al (ere lu o) Coles -sssol som te LAN, 
A Doesné distinguish organizational We) cer-lalyzelilolaral ie leoleascmel-vilalitiolamvsy-ve 
standard and project processes to capture best practices 


A Emphasis on training to motivate and A Emphasis on infrastructure to ensure 
communicate skills key processes addressed 


A Reliance on statistical methods to A Reliance on statistical methods to 
airelarelelom el-)a(e)anarelalerc) anrelarelelom el-)a(e)aanrelalec) 
A Focus on learning from internal A Additional mechanisms leverage 
experience and data external technology 
A Prioritization of efforts based on PN a) ancolesiegelcato}(om o)f-lalaliaremuvstel 
business payoff improved with V1.2 
A Certification of individual practitioners A Certification of assessors and 

(o) fefelal r4euile) ars) 


Adapted from Card, Integrating Lean, Six Sigma, and CMMI 


al 


Opportunities for synergy 


A CMMI provides a set of best practices for program 
management, systems and software engineering, 
Valo acto] e-1=telm e) mole] Ulormr-lalem eo) geler-\-t-me(-\/-)(0) e]aat-Jals 

A CMMI provides a measurement scale to benchmark 
organizations against each other 

A CMMI defines requirements for measurement and 
statistical analysis Lean Six Sigma provides tools to 
implement 

A Lean Six Sigma provides a process for performing 
‘ol oXer-\-t-wr- Lalo e)weole(Uleim inal elceh(-lial-valm-leehuia(ss-) 

A Lean Six Sigma provides a strong mission business 
rol at-Valecl ite) am com e)oler-\-t-mlanle) cen -lait-vals 


ean Six eee hae C IMIVII 

impler Lifeyal 

A Provides a tactical engine for continuous 
Taal dxed {teat salmice)aom )eeovol(-usmel-Vilaliirolamislcolereial 
Tan) eLCeVaalsvalecidkelameymerar-lale [= 

A Supports rational decision-making through 
credible measurement of benefit 

A elps solve specific problems and improve 
J of =Yed} i (om 0) ole | U Lore me) me) geler-\-t-1-s-M Vidal iam daly 
Flue l=¥ Moxey alt=) qe) melael-lalyc-lilelar-] me) meler-s-1-) 
improvement 

A Requires alignment with business drivers 

A Garners effective sponsorship for change 


Using LM2 and CMMI Together 7; 


A Within Lockheed Martin, LM2 is being used to 
JU] 0) of) a mt OF\'/1\"| Mumm of: t-{-10 me) weler-\-t-{-.— 
- Implement CMMI Processes 
- Improve CMMI Processes 
- Achieve CMMI _ igh Maturity 
- Improve the Improvement Process 


Use LM2 to Implement 
CMMI Processes 


Final Audit Brief Publish report to: | Finalizeaudit banal | | Stk 


Process owner 

#Process ownerés 
Discuss findings manager y [=] H 
and Program Manager : i 
recommendations sIBP 0807 People ——— 

sin ‘i ‘nin ‘nda Lim: 
; ES 
Schedule follow Brief Directors 
up verification Status 
‘nin 
“AD 


ae HEEL 


LUTTE TT 


approved 
by audit 
ier? 


A Use Lean principles to improve processes to 
to improve processes and six sigma to 
measure results 


A Drive process improvements where 
quantitative measures show actions are 
needed 

A When LM2 is used with CMMI , programs see 


meaningful, measurable performance 
improvements : 


- 
improvement gq = 
x 10.0 - 
. . . g a y= 
in defection detection Ae 
w 40 { | ot = 
20 =o 
. . a SS 
ig=Yo[U lexi (eye In Halegel || Peale cUuT Phase 2 Phase3 | swreqver | Seg Ver SEIT 
Design Design 
| Projected 00 00 0.0 0.0 0.0 0.0 
cos (o} IX e Cc Ss = Actual Defect Density 82 188 25 30 05 07 
“Complete 
Estimate 


Use LM2 Critical Skills 
to Achieve CMMI — igh Maturity 


1p 


g 
= 
3 
6 


Miscellaneous 

Initialization 

User_Interface 
Literal_Use 


Collateral_my 


(Sibcatecon] 


Use LVi2 to Improve the 


improvement Process 


A LM2 can be used to identify the highest priority 
elo) o)(=Ta atm com-1e) o) elolam lat) e)cedv(-ial-lalm ele) (-verme-y-(-lerd(o) a) 
rVaYow okelanve)i(omast-larele(-yaat-vals 
- Program Excellence Plans (PEPs) link strategic and 
tactical goals with process improvement initiatives 
A Optimize improvement program execution 


A Common repository provides a record of the results of 
all improvement activities 


Project to lean improvement Pate 
e) kefer=\-f-] 


integrate Improvement Initiatives 


ALM2 and CMMI integrated into an 
ro) er-lalyezclitelaruite(=mefed(-1aal-lalet-maarevel-| maar- 1s 
provides a standard for program execution 
A Establish a set of standard processes that 


incorporates all the features of the initiatives 
ro) med aLoy (or) 


Process flow for 
. 
using LM2 for causal 
- 
= | analysis 
O issut 
,| Gather 
Data 
08 i—® © @ 
a Identify | == —‘| Perfor m rioritiz Identify 
& + Possible |} Causal tential 
— ‘| Causes Analysis. auses Solutions 
i—_@ iO) © O 
Develop] =| imntement | [Communicate 
® ” fee ener | *] Acknowledge 
g @ |_ Success 


Conciusions 


A Within Lockheed Martin, Lean Six Sigma and 
CMMI have demonstrated their value in the 
LUT g-JU] mo) mm ol: ai-vedm olgetele-luim el-l ace) atar-lales:) 


A Lean Six Sigma brings focus on business 
needs and measurable improvement to CMMI 
based improvement 


A For those responsible for rolling out process 
improvement efforts - design your 
implementation so Lean Six Sigma and CMMI 
interoperate 


Card, David N., “Integrating Lean, Six Sigma, and 
CMMI.” http vc.ieee.org.ar 2 -Card- 
Taitctele-idlate by m=r- 1am eleli 


Siviy, Jeannine, Penn, M. Lynn, arper, Erin. 

“Relationships Between CMMI and Six Sigma.” 

CMU SEI-2 -TN-  , December 2 

alid oem YA AN x2) og sa LUM tel Om elele)|(er- life) at-melelerelaat-1 ai c-) 
reports tn tn =. html 


The Potential for Lean Acquisition 
of Software Intensive Systems 


Software Engineering Institute 
Carnegie Mellon University 
Pittsburgh, PA 15213 


Jeffrey L. Dutton 
Acquisition Support Program 
14 November 2007 
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accomplishes its tasks with a cyclic rhythm that responds to the needs 


of its customers and stakeholders. Under lean 
manner, removing defects at the point of inception. The Chief Engineer 
owns the system architecture, and ensures that the activities of the 
highly skilled program staff are) architecture-centric. 
are improved quickly. The improvement| infrastructure is lean, CMMI- 
and process control 

in their proce (which 
, and 


The lean program office team is fast, flexible, learns quickly} and 
responds to change quickly, using mature. controlled processes that 
compliant, and uses six sigma mechanisms 
aoporoach ne staff has eliminated waste é 
are integrated with key contractor and stakeholder processes) 
Carnegie Mellon 


Software Engineering Institute 


The Lean Program Office..... 
The lean program office is sharply focused on customer value, and 
prodram Managem 
simultaneous teams accomplish their work in a highly synchronized 


33 
(to Mr. Sakamoto -“Gung 0”- ) 


“What, are you crazy 


Some outrageous claims followé . 


Lean Thinking has not bDe)~ sea 
A Lean is MUCH mo) i : 


A The full context of 
Acquisition domait 


ironments is applicable to the 


The CMMI is a KEY ENA tn Thinking in creative 
environments ¢ 


Together, Lean Thinking 
the most stressing proble 


solution space to some of 


Lean Thinking, the \G ses) Sigma are perfectly 
Popeetien es, that, together, 


interactive, mutud| i 
provide an impro ) . yaradigm that is 
than any of them alone. 
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Lean CMMI Sixo enn Diagram 


Almprovement framework 
AMature best practices 
ARobust appraisal methods 


AStrategic focus 
ACustomer value 
ARapid improvement 


Lean Thinking 
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Experience in Lean Software Development 


Application of Lean Software Development ina CMMI-DEV 
compliant environment has led to: 


A Higher productivity 

A Reduced product defects 

A Much faster cycle times/product through-put 

A Integration of project performance with CMMI capability or maturity 
A Faster learning teams and organizations 

A Absolute solution to fbuy-indissues 

A (Much) faster improvement of processes and performance 


Software Engineering Institute | Carnegie Mellon 


The Real Content.... 


Software Engineering Institute | Carnegie Mellon 


Outline 


Background: 
A What is Lean for creative environments about? 
A Why is the CMMI a valuable flean enabler? 
What is the Lean Program Office? 
A What it looks like 
A How it might operate 
How can we get there from here? 
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Lean Thinking for Creative Environments 


Sharp, continuously refreshed focus on customer value 


Iterations and synchronization (racquisition cadenced 


rAgileOprogram management 
Highly skilled individuals 
Highly developed teams 


Rapid improvement 


Waste elimination 
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The CMMI is a Lean Enabler 


Mature set of domain-specific practices 
A System, software, and hardware development 
A Acquisition 
A Lean interpretation is now an accepted practice 
6 Required elements (goals) 
6 Expected elements (practices) 
Proven infrastructure for process management 
A Responds to observed lack of lean process mot. 
A Multiple VSMs may lose data and process workflow/process interfaces 
Accommodating appraisal method 
A SCAMPI is a well respected method with three value/ROI propositions 


A SCAMPI supports assessment of lean objectives and lean alternative 
practices 
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Lean Application of the CMMI-AC 


16 core process areas 

6 Acquisition-Specific process areas 

Goals are all REQUIRED elements of the model 
Practices are all EXPECTED elementsé 


A Alternatives (for practices that may not support lean operations) may be substituted 
(even many-for-one if appropriate) 


A The Goal must always be supported 
Extensive experience base in fheaningOCMMI practices 


In the end, CMMI-ACQ will be a high-value Lean enabler for the Lean Program 
Office 
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CMMI-AC_ Process Areas 


Agreement Management 
Acquisition Requirements Development 
List of Core Process Areas Acquisition Technical Management 
Acquisition Validation 
The following processes are considered core in the CMM Acquisition Verification 


Foundation: : : 
Causal Analysis and Resolution (CAR) ———————______ iia iel jaerneloalenicteanaaui 


Configuration Management (CM) —J4J2£2_—_————_ Configuration Management 
Decision Analysis and Resolution (DAR) ———____________ Decision Analysis and Resolution 


Integrated Project Management (IPM) ————______ : 
Measurement and Analysis (MA) Integrated Project Management 


Organizational Innovation and Deployment (O Measurement and Analysis 


Organizational Process Definition (OPD ‘Sati : 
Organizational Process Focus (OPF) Organizational Innovation and Deployment 


Organizational Process Performance a Organizational Process Definition 
Organizational Training (OT) — Organizational Process Focus 
Project Monitoring and Control (P ae 

Project Planning (PP} Organizational Process Performance 
Process and Product Quali ance (PPQA Organizational Training 


Sebring nolianbas pray Project Monitoring and Contr 
Risk Management (RSKM) Project Planning 

Process and Product Quality Assurance 
Quantitative Project Management 
Requirements Management 
Risk Management 
Solicitation and Supplier Agreement Development 


U 


. . . . . . . . . * * . . . . . 
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Lean Application of CMMI Practices (Examples) 


Technical Reviews 


eee ee ee ee ee ee ee ee ee ee ee ee ee ee ee ee ee ee ee ee ee ee ee ee ee ee ee ee ee ee ee ee ee ees 


Initial Technical Review (ITR} 

Alternative System Review (ASR) 
Integrated Baseline Review (IBR} 
Technology Readiness Assessment (TRA) 


System Requirements Review (SRR} =F: Ta celeice| suggest 
System Functional Review (SFR) Creative iterations 


Preliminary Design Review (PDR) Continuous test and defect removal 
Critical Design Review (CDR) TUE) maatsiialess 
Test Readiness Review (TRR)} 


ola) oy: Tame (stere-J lel ary 
Possibly synchronous teams 


System Verification Review (SVR) 
Production Readiness Review (PRR) 
Operational Test Readiness Review (OTRR} 
Physical Configuration Audit (PCA) 
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Lean Application of CMMI Practices (Examples) 


Risk Management 


SG 1 Prepare for Risk Management 
SP 1.1 Determine Risk Sources and Categories 
SP 1.2 Define Risk Parame 


SP 1.3 Establish a Risk Mang W=y-Ta Moti fe Ure fe (=x-15 
SG 2 Identify and Analyze Risks Rapid mitigation 

SP 2.1 Identify Risks Keim colerel-melamanlid(er-ii(elamyc-y(eleq tay, 
Integrated risk management 


SP 2.2 Evaluate, Categorize 
Predictable Risk Mgt. work cycles 


SG 3 Mitigate Risks 
SP 3.1 Develop Risk Mitigatfe 
SP 3.2 Implement Risk Mitigation Plans 


Software Engineering Institute | Carnegie Mellon 


Lean Application of CMMI Practices (Examples) 


Process and Product Quality Assurance 


SG 1 Objectively Evaluate Processes and Work Products 
SP 1.1 Objectively Evaluate Processes 


SP 1.2 Objectively Evaluate VVork Products and Services 
SG 2 Provide Objective Insight 
SP 2.1 | Communicate and Eng My Tafel el Ce RLU fe fe [--33 
SP2.2 Establish Records SWF atedal we) alelet-w-leceli men cer (ct) 
a€-laley-lamareliiicer-ii(e)atcw-lace mmm el0i| aar-Lece lic 


Prepared teams, rapid audits 
DY =)(:Yoac-wol-Jalaii(crem-larem-liieiliarciccem ial 
days, not weeks or months 
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What is the Lean Program 


@hiiler=. 
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= VISUAL alCiileClures are Meecued 


What is the Lean Program Office 


Fast learning and early maturity Fast learning and early maturity 
Led by an Agile Program Managed by an Agile Program Manager 
Architecture-Centric Operations Architecture-Centric Operations 


DoD Architecture Framework 


oe a. 


rocess Integration and Synch. 


anban visual Decisions 


DeCISIOAS ssc 
= Mature Lean PMO eee tightly 


NKFARACCAC 


7 Visual metrics or indicators fpushedo 
to right person at right time 

= NO mgt. rdirectiono 

" Activities are Mulledoby PMO team 
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What Makes Lean Work 


Constant focus on customer value 
Waste elimination 
Shared vision/architecture Kaizen Team should OWN the process 
Concurrency cice of the Customer 

Parallel cohesive activities Gg ay” 

Information flow 


Iterations and synchronization 
Agile Project Management 
Rapid learning 

Skilled teams 

Rapid improvement 

Process ownership by the process fdoero 
Visualization and Kanban actions 


alue Stream Mapping 


Continuous Waste Elimination 


Mgt should NOT change the process 
except by going through Kaizen team 
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What Makes Lean Work 


Constant focus on customer value 
Waste elimination 
Shared vision/architecture When pointer enters yellow area, 
Mgt. Team takes a specific, immediate action 
Concurrency 
Parallel cohesive activities 
Information flow 
Iterations and synchronization 
Agile Project Management 
Rapid learning 
Skilled teams Eliminates much “wait time 
and mgt. waste 
Rapid improvement 
Process ownership by the process fdoero 
Visualization and Kanban actions 


Enterprise or program dashboards 
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What are Acquisition Iterations ( of2) 


Source Selection 
A REP generation and evolution 
A Proposal evaluation 
Program Execution 
A Architecture evolution (creative iterations) 
A Oversight of contractor team (operational cycles) 
6 Contractor process capability 
6 Requirements mgt. 
6 Risk mgt. (lean risk mgt.) 
6 Budget oversight 
A Oversight of technical deliverables 
6 Depends on developer life cycle (Single step, incremental, spiral SW) 
6 Preparation for technical evaluation 
6 Technical evaluation 
6 Feedback to developer in a timely manner 
PMO Team development 


To 


A Communications with stakeholders 
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(2 of 2) 


What are Acquisition Iterations 
takt time drivers 
P-Noxe[Uits}lile)a mere \e(clalercurelalemc\Viarealee)aiyaqlilela 
eUICO) VSM UaTomcd il pallarchile) ame) meatomie)ile\niiale may elots 


| of waste: 
= Wait time and work product decay 


f= Motion 
=" Unused work products 


= Task switching 
Systefn Requirement$ 
ponnement : : 
Software Increment 


Carnegie Mellon 


System Development 
Software Increment 
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Determining the Takt time... (added slide) 


Takt Time: the rate that a completed product needs to be finished in order 
to meet customer demand. May be thought of rcadenceoor rmheart beatoof 
the organization 


Drivers for Takt Time: 
o Customer/user demands 
o Congressional demands 
o fSystem of systemsoor rcommon systemodemands 
o Demands from other stakeholders 
O 


Development team processes and schedules 


Software Engineering Institute | Carnegie Mellon 


Lean Teams 


Agile Program 
Manager 
Chief Architect 


Highly skilled 


team memes ZA a 


Communicate |R= 
openly 


Agile Engr. 
Support 
(CM/QA/Test) 


Carnegie Mellon 


== Software Engineering Institute 


Lean Organizations 


makevile(omesle)(elarcr-val 


improvement 
Talinctsviaeleielas 


Ul arolcyacire late e)celecisys 


reVale im tcxeva] ave) (el@ hv, 
(ol isyorl e)i fate 


PN CoNViVan (otelamielalersle)ats 


relalomccr- lancome) ia 
their processes 


Stimulate 
o)cerelalyzelile)ary 
KereTaaliate 


Focus on customer- 
ol aN elam el-sace)anerelarere 
objectives 


Software Engineering Institute 


Carnegie Mellon 


rol! Blown 's)(-WC(-\ am al: la-Miaelan 


ere 
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ow do we get There from. ere 


Perform Value Stream Mapping session for core PMO as early as possible 
A Define PMO performance measures 
A ldentify near term fiuture stated 

Add (lean) CMMI-ACQ as early as possible 


Mentor lean/CMMI team members to develop tacit knowledge as quickly 
as possible 


Adopt lean/CMMI processes based on workflows 
Work toward a Lean CMMIOPMO capability: 
A Team Level 2 by RFP release? 


A Team Level 3 by contract start? 


Software Engineering Institute | Carnegie Mellon 


For more information, contact: 


Jeffrey L. Dutton 
iid@sei.cmu.edu 
or 

Brian Gallagher 


bg@sei.cmu.edu 


Software Engineering Institute (ASP) 
4500 Fifth Avenue 
Pittsburgh, PA 15213 


Software Engineering Institute | Carnegie Mellon 
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Bogen oy Features 


SCRUM is based on the principle that software development is an 
empirical process, not a defined process, and that if you try to 
manage an empirical process with a system designed for defined 
processes, you are doomed to fail. 


= SCRUM fexpects the unexpectedoand fcontrol is exercised through 
frequent inspection and adaptationo 


= SCRUM is a borrowed term from fRugbyo 


= It claims to be CMMI Maturity Level 3 compliant 
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Product Backlog sprint Planning 
(Prioritized To Do List) 


| 
ne , Scrum 


Product Owner 
(VISION!) 


Reflection 


Selected Product 
Backlog 


Flow. 


wr, 


New Functionality 


sprint Planning 2 


Sprint 
Daily and Monthly 


An ED ED 
PADDED 
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ECOMPIete en Comat 


Phases 


emeed Features 


Planning 


m Initial requirements 

= Initial release planning 

= Architectural and business vision 

= Establishment of ROI gauges with initial funding 


Staging 
= Addition of non-functional requirements to project foacklogo 


= Route maps 


Development 
m 30-day fsprintso 


=m 1 day bursts of implementation 


« ™ Release 

S =m Enough usable functionality is developed, or AL ED ED 
5 = Release date is achieved FPAIDEID 
© 
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use period has ended, | 


Complete ircomaets_ 


= | 


Establish Estimates 


Product backlog prioritized product features Top level WBS: work packages, task 
desired by the customer descriptions 


Sprint backlog (30 days) tasks to create Evolutions can be also considered in the 
Increment. Lifecycle is evolutionary. definition of the project lifecycle 


Estimates are established for sprint backlog Estimates are determined for complete WBS 
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Complete "nsrcomaete 


Develop a Project Plan 


Saeed Features 


Budget is agreed at the beginning of the Establish budget and schedule and maintain it 
project and revised at each iteration. Product 


Owner manages ROI; business decision at 
the end of the sprint on project continuation. 
Sprints deliver results every 30 days 

Daily Scrum meetings held every 24 hours 


Project risks are analyzed and evaluated Identify project risks 
daily 


Project data primarily includes product Plan for management of project data 
backlog, sprint backlog, product increment 


Project resources discussed at Sprint Plan for project resources, knowledge and 
Planning Meeting and Daily Scrums skills, and planning for stakeholder involvement 


Product backlog and sprint backlog act as 
o the project plan Establish and maintain the overall Project Plan 


AL ED ED 
PADDED 
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Obtain Commitment to the Plan 


feed Features 


Product backlog and sprint backlog act as Review all plans that affect the project 
the project plan 


Sprint planning reconciles resources. Daily Reconcile work and resource levels 
Scrum meetings revises resource needs 


Plan commitment obtained at Sprints and Obtain plan commitment 
Daily Scrums; team as a whole owns the 
sprint plan, and commits to meet it 


© ABB, 2007} 7 


Your complimentary 

use period has ended. 
Thank you for using 
PDF Complete. 


Monitor Project Against Plan 


feed Features 


Use backlog to monitor progress and Monitor project planning indicators of project 
commitments. progress 


olunteerism used as work load balancing Monitor commitments 


Discuss risks and issues and remove Monitor project risks 
impediments at Daily Scrum meetings 


Monitoring done at Daily Scrum and Sprint Monitor data management 
planning meetings 


Stakeholder involvement monitored at Daily Monitor stakeholder involvement 
Scrum meetings 


Progress and milestone reviews conducted Conduct progress and milestone reviews 
during the Daily Scrum and Sprint planning 
meetings 
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anage Corrective Action to Closure 


Done at Daily Scrum and Sprint planning Analyze issues to determine corrective actions 
meetings 


Corrective actions taken and managed to Take corrective actions on identified issues. 
closure primarily at Daily Scrum meetings Manage corrective actions to closure 
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m “ ome Ground” 


Milemaseeey) i) Features 


Meeting highly volatile customer needs (requirements) 


= Managing tighter dependencies among team memberG 
work 


= Having co-located teams 
= Coping with shorter planning horizons 


= Building Hust enoughosoftware functionality, and then 
ending the project 


= Having a small development team 
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Effective Systems Engineering: 
What’s the Payoff for Program 
Performance? 


NDIA Systems Engineering 
Effectiveness Committee 


CMMI Technology Conference 
November ,2 


NDIA Systems Engineering Effectiveness 1 
November 15, 2007 


use period has ended. : 
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The SE efforts on my project are 
critical because they ... 
... pay off in the end. 


... ensure that stakeholder 
requirements are identified and 
addressed. 


... provide a way to manage 
program risks. 

... eStablish the foundation for all 
other aspects of the design. 


... optimize the design through 
evaluation of alternate solutions. 


We should reduce SE efforts on this 
project because ... 


... Including SE costs in the bid will 
make it non-competitive. 

... we don’t have time for ‘paralysis 
by analysis’. We need to get the 
design started. 

... we don’t have the budget or the 
people to support these efforts. 

... It doesn’t produce deliverable 
outputs. 


... the customer won’t pay for them. 


These are the ASSERTIONS, but what are the FACTS 


NDIA Systems Engineering Effectiveness 
November 15, 2007 
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It is difficult to justify the costs of SE in terms that program 
managers and corporate managers can relate to. 
A The costs of SE are evident 
- Time 
- Effort 
A The benefits are less obvious and less tangible 
- Cost avoidance (e.g., reduction of rework from interface mismatches 
- Risk avoidance (e.g., early risk identification and mitigation) 


- Improved efficiency (e.g., clearer organizational boundaries and 
interfaces) 


- Better products (e.g., better understanding and satisfaction of 
stakeholder needs) 
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eering Effectiveness Survey 


STRENGTH THROUGH INDUSTRY iCHNOLOGY 


ypothesis The effective performance of SE best practices ona 
development program yields quantifiable improvements in the program 
execution (e.g., improved cost performance, eeneoule pevomene 
technical performance). : 


e 0 
File Edit Yiew Favorites Tools Help 


Qe ball “> Jee |x] 2] Hit ) Search 2 &)| Yate = jw] ~ Wie 


Objective Ss Address - edu/feedback/SystemsEngineering_Demo.htm 
A Characterize effective SE practices a 
. . Software a Institute ae. oe 
A Correlate SE practices with measures The Effectiveness of Systems 
of program performance Engineering: A Survey 
Approach ee ee ee ~"E 
A Distribute survey to NDIA companies oe ee ee 


© Estimated 


ASEI analysis and correlation of responses cea 


wf 


LL ff 1B [@ internet Y, 


Survey Areas 


Process definition Trade studies Project reviews 

Project planning Interfaces Validation 

Risk management Product structure Configuration mgmt 
Requirements development Product integration Metrics 

Requirements management Test and verification 
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Previous Studies - Summary 
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STUDY APPLICABILITY 


Author & es eotaere 
Findings SE Activities 
Background 
Gruhl (1992) 8-15% Upfront ]| First two of five 
32 NASA Pgms 


Best development phases 
Herbsleb (1994) 


Process 
13 CMM CMM Process 
Areas 


Improvement 
Companies ROI 4.0T 8.8 

Honour (2004) 15-20% of Overall SE level of 
Survey INCOSE project should effort (Cost) & 

SEs be SE related SE quality 
Boehm & Valerdi | SEimportance J— COCOMO II RESL 
(2006) grows with (Architecture and 
COCOMO II Risk) 


project size 
: Estimate 

Boehm & Valerdi 

(2004) 


within 30% 33 activities defined 
COSYSMO 


effort 50% - by EIA 632 
70% of time 

Ancona & 

Caldwell (1990) 

Boundary 


Management 


Team boundary 
activities T interface 
between team and 
external 


Managing team 
boundary 15%; 
more is better 


More SE 
yielded better 
quality & 
shorter 
duration 


Frantz (1995) 
Boeing side-by- 
side projects 


Defined by Frantz 
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Definition of | Characteristics 
Success of Project 
Cost (Less cost Large; Complex; all 
overrun) NASA 
eee Gort Various; federal 
contracting 


reduction through 
Various sizes 


SE investment) 
(measured by total 


Cost & Schedule 
Various sizes, but 
Cost 


software systems 


only 
Product 
Performance 
(Successfully 
marketed products) 
Product 
Performance & 
Schedule (Quality 
of product and 
duration of project) 


Mostly successful 
projects from 
federal contractors 


Technology 
products 


Three similar 
systems for 
manipulating 
airframes during 
assembly 


Mink, 2 
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Total Program Overrun 


32 NASA Programs 
Definition _ Definition$ 
Percent ~~ Target +Definition$ 90% Assurance (1.6 c) 

c Program _ Actual + Definitions 222 

S Overrun Target + Definition$ oO Average Cost Overrun 

: : 
Oo 51.8 

E = 

is) oS 

= 2 

} a 

—_ 
ou 

1.0 —*—— 
0% 4% ~~ 8%: -- 912% _ 16% 20% 24%... -28% 
10 15 
Definition Percent of Total Estimate SE Effort = SE Quality * SE Cost / Actual Cost 

Gruhl, Werner (1992), Lessons Learned: onour, Eric (2004), Understanding the Value of 

Cost/Schedule Assessment, Internal Presentation, Systems Engineering, Proceedings of the 14" Annual 

NASA Comptroller@ Office INCOSE International Symposium 
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A 14 Process Areas 
A 31 Goals 

A 87 Practices 
A199 Work Products 


CMMI-SE SW IPPD 
v. Systems 


Engineering- 


| = = 
related Filter 
-“ fi 


Size Constraint 
Filter 


A 25 Process Areas 
A179 Goals 

A 614 Practices 

A 476 Work Products 


A13 Process Areas 
A23 Goals 

Aa45 Practices 

A71 Work Products 


Survey content is based on a recognized standard (CMMl) 
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Survey 
Population 


Sampling Method 


Survey 
Deployment 


Target 
Respondent 


Questionnaire 
Structure 


Target Response 
Time 


Responses 


Analysis 


Reports 


oO d O Oo g y NATIONAL DEFENSE INDUST: TON 


Organizations developing products in support of government 
contracts (prime or subcontractors). 


Invitation to qualifying active members of NDIA Systems 
Engineering Division. Random sampling within organization. 


Web deployment (open August __, 2 -November ,2 
Anonymous response. uestions based on CMMI-SE SW IPPD Vv. 


Program Manager or designee(s) from individual projects 


Characterization of the project program under consideration 
2. Evidence of Systems Engineering Best Practices 
Project Program Performance Metrics 


= minutes 


survey responses (complete partial, but usable) 


Raw data analyzed by Software Engineering Institute. 
Analysis results reviewed by NDIA SE Effectiveness Committee. 


. Public NDIA SEI report awaiting approval. 
2. Restricted attachment, details provided to respondents only. 
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Perf f(PC, PE, SEC, AC) 


where: Perf = Project Performance PC = Project Challenge 
PE = Project Environment AC = Acquirer Capability 
SEC = Systems Engineering Capability 
SEC can be further decomposed as 
A Project Planning 
A Project Monitoring and Control 
A Risk Management 
A Requirements Development and Management 
A Technical Solution 
- Trade Studies 
- Product Architecture 
A Product Integration 
A Verification 
A Validation 
A Configuration Management 
A IPT-Based Capability 


SE capabilities and analyses are fully defined by mappings of 
associated survey question responses 
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Maximum = 2.8 
3" Quartile = 2.1 
Median = 1.9 

1 Quartile = 1.7 
Minimum = 1.1 


Maximum = 3.9 


3" Quartile = 3.3 
Median = 3.0 

1* Quartile = 2.7 
Minimum = 2.1 


N=63 N= 64 


Overall SE Capability (SEC) Project Challenge (PC) 


Maximum = 4.0 
3™ Quartile = 3.1 


Acquirer Capability (AC) 


Maximum = 4.0 
3" Quartile = 3.1 
Median= 2.8 
1*t Quartile = 2.4 
Minimum = 1.5 
N= 64 


Project Performance (Perf) 


Analyzed distributions, variability, relationships... 


To ensure statistical rigor and relevance 


Median = 2.75 
1*t Quartile = 2.3 
Minimum = 1.7 
N= 46 
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ity (SEC) 


Best 
Performance 
(x > 3.0) 


Moderate 
Performance 


25<xs3.0) 


Lower 
Performance 
(x < 2.5) 


Lower Moderate Higher 

Capability Capability Capability 
(x $2.5) (2.5<x<3.0) (x 23.0) Gamma = 0.32 
N= 13 N=1? N= 16 p= 0.04 


NATIONAL DEFENSE INDUST: 
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Notation 


Projects with better Systems Engineering Capabilities deliver better 


Project Performance (cost, schedule, functionality) 
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Project challenge 


factors 

Aife cycle phases 

Project characteristics 
(e.g., size, effort, 
duration, volatility) 

echnical complexity 

Teaming relationships 


Challenge 


re 
8 
w 
% 
ra 
i 
= 
5 
a 


Lower Moderate Higher Chall 
Capability Capability Capability ene 


Projects with better Systems Engineering Capabilities are 


better able to overcome challenging environments 
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. Product Architecture and Performance Sear eee 


Best 
Performance 


(x > 3.0) 


Moderate 
Performance 


(254x230) 


Lower 
Performance 


(x < 2.5) 


Lower hloderate Higher 

Capability Capability (Capability 

Me 24 (272 e = 3 ME S34 Gamma = 0.40 
M=18 M=14 M=13 b= 0.002 


Projects with better Product Architecture show a 
“Moderately Strong Strong” Positive Relationship with Performance 
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Best Performance 
(x > 3.0) 


Moderate 
Performance 
25<xs3.0) 


Lower Performance 
(x < 2.5) 


Lower Moderate Higher 

Capability Capability Capability 

xs 2.7) (2.7 <x< 3.3) (x= 3.3) Gamma = 0.37 
N=18 N=12 N=16 p=0.03 


Projects with better Trade Studies show a 


“Moderately Strong Strong’ Positive Relationship with Performance 
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> 


Best Performance 
(x > 3.0) 


Moderate 
Performance 
(2.5<xs 3.0) 


Lower Performance 
(x < 2.5) 


Lower Moderate Higher 

Capability Capability Capability 

(x= 2.8) (2.8 <x < 3.2) (= 3.2) Gamma = 0.36 
N=15 N=15 N= 15 p= 0.03 


Projects with better Technical Solution show a 


“Moderately Strong” Positive Relationship with Performance 


NDIA Systems Engineering Effectiveness 15 
November 15, 2007 


NATIONAL DEFENSE INDUSTRIAL ASSOCIATION 
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Best Performance 
(x > 3.0) 


Moderate 
Performance 
25<xs3.0) 


Lower Performance 
(x < 2.5) 


Lower Moderate Higher 

Capability Capability Capability 

(es 2.5) (2.7 <x <3.1) (R= 3.1) Gamma = 0.34 
N=15 N=16 N=15 p= 0.04 


Projects with better IPTs show a 


“Moderately Strong” Positive Relationship with Performance 
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. Requirements and Performance awaer eee 
>| 


Best 
Performance 
(x > 3.0) 


Moderate 
Performance 


254" 25.0) 


Lower 
Performance 
(x < 2.5) 


Lower Moderate Higher 
Capability Capability Capability 
ie = 2.8) (ben 2 3.4) 23.4) Gamma = 0.33 
a 


M= 1 M=19 M=14 b= 0.04 


Projects with better Requirements Development and Management show a 


“Moderately Strong” Positive Relationship with Performance 
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Summary of Process Relationships eee 
Details Relationship of SE Processes to Program Performance 
>| Architecture 
>| Trade Studies 
>| Technical Solution 
>| IPT Capability 
| £  ReqtsDevel Mgmt 
> 3 alidation 
my és Risk Mgmt 
Bs; UW erification 
>| Product Integration 
>| Config Mgmt 13% 
>| Project Planning ===sy 13% 
>| Project Monitor Control | -13% 
-20% -10% 0% 10% 20% 30% 40% 50% 
Gamma (strength of relationship) 
i Strong Relationship a eae Nae | see fc || Weak Relationship 
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i Reqts Tech Solution: 


Relationship of SE Processes to Program Performance 


ee ciares Architecture 40% 
Trade Studies 37% 


Technical Soluti 


IPT Capability 134% 


Reqts Devel Mg 


alidati 

Risk Mg 

erificati 

Product Integrati 
Config Mg 

Project Planni 
Project Monitor Cont 


on L 136% 


mt : 133% 


on 128% 
mt 28% 
on 25% 
on 121% 
mt 13% 
ng 13% 
rol | -13% 


A9% 


63% 


-20% -10% 0% 10% 20% 30% 40% 


Gamma (strength of relationship) 


50% 


60% 


70% 


[Bl strong Relationship UI to Strong Relationshi 


Moderately Strong a Moderately Strong 
Relationshi 


[ | Weak Relationship 


NDIA Systems Engineering Effectiveness 


November 15, 2007 


19 


Your complirr 
use period has 
Thank you for 


Project challenge 

factors 

ALife cycle phases 

Project characteristics 
(e.g., size, effort, duration, 

volatility) 

Technical complexity 

ATeaming relationships 


Projects with higher 


Lower Challenge 
(PC £1.9) 


Moderate 
REQG+TS 
Capability Capability 
(2.8¢x<e3.1) (23.1) 
N=6 N=7 


Lower 
REG+TS 
Capability 


Higher 
REQ +TS 


Gamma = 0.57 
p-value = 0.02 


Requirements and Technical Solution 


Higher Challenge 
(PC >1.9) 


Moderate 
RE@+TS 


Lower 
REQ@Q+TS 
Capability Capability 

(%S2.8) (2.8¢x<3.1) 

N=7 N=7 
Gamma =0.54 
p-value =0.03 


Higher 
REQ@+TS 
Capability 
(x23.1) 

N=11 


STRENGTH THROUGH INDUSTRY & TECHNOLOGY 


Best 
Performance 
(x > 3.0) 


Moderate 
Performance 
(254x4£3.0) 


Lower 
Performance 


(x = 2.5) 


capability are 


better able to achieve higher performance even in challenging programs 
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eee Features 


SE Effectiveness 
AProvides credible measured evidence about the value of 
disciplined Systems Engineering 
AAffects success of systems-development projects 


Specific Systems Engineering Best Practices 
AHighest relationships to activities on the fleft side of SE Veed 
AThe environment (Project Challenge) affects performance too: 
- Some projects are more challenging than others ... and 
higher challenge affects performance negatively in spite of 
better SE 
- Yet good SE practices remain crucial for both high and low 
challenge projects 
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Vext Steps 


Provide recommendations for action upon survey findings 


Conduct additional follow-on surveys and analysis of 
collected data 

AIV&V 

ABroadened sample space 

ATrending 

Almprovements to survey instrument 


Survey system acquirers 
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Backup 


NDIA SE Effectiveness Survey 
Analysis Slides 
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Caveats 


Consistent with “Top 10 Reasons Projects Fail*” 


. Lack of user involvement 
2. Changing requirements 

Inadequate Specifications 
Unrealistic project estimates 
Poor project management 
Management change control 
Inexperienced personnel 

. Expectations not properly set 

. Subcontractor failure 
.Poor architectural design 


Project Management Institute Matching items noted in RED 
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Caveats 
Consistent with “Top 5 SE Issues*” (2006) 


AKey systems engineering practices known to be effective are not 
consistently applied across all phases of the program life cycle. 


Alnsufficient systems engineering is applied early in the program life 
cycle, compromising the foundation for initial requirements and 
architecture development. 


ARequirements are not always well-managed, including the effective 
translation from capabilities statements into executable requirements to 
achieve successful acquisition programs. 


AThe quantity and quality of systems engineering expertise is insufficient 
to meet the demands of the government and the defense industry. 


ACollaborative environments, including SE tools, are inadequate to 
effectively execute SE at the joint capability, system of systems, and 
system levels. 


OUSD AT L Summit Matching items noted in RED 
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istogram of 
response 
frequencies 
Median 
Maximum = 3.8 
3” Quartile = 3.2 
Median = 2.8 
1* Quartile = 2.4 
Minimum = 1.0 
N= 64 
Interquartile 
Outliers Range 
Y~ Sample size 
Data (responses to corresponding 
Range survey questions) 
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Mosaic Chart Var eee 


lal Column width 
represents proportion 
of projects with this 
level of capability 


Best Performance 


(x > 3.0) 
Moderate . 
Baifarnaciié Relative performance 
(25<x<$3.0) distribution of the 
sample 
Lower Performance 
(x < 2.5) 
Lower Moderate Higher 
Capability Capability Capability 
s25)  (25<x<3) (x= 3) Gamma = -0.13 Gamma Nees Se strength of 
N=13 N=13 N= 20 p= 0.25 relationship between two ordinal 
variables 


p probability that an associative 
relationship would be observed 
by chance alone 


Projects exhibiting a given 
level of relative capability 
(Lowest, Intermediate, ighest) 
Sample size and distribution for association 


associated survey responses and statistical test 
(capability performance) 


Measures of 
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Best Performance 
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Performance 
2.5<xs<3.0) 
Maximum =4.0 
ee aa ia Lower Performance 
F 1* Quartile = 2.6 (x < 2.5) 
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N=18 N=14 N=13 p= 0.002 
Relationship to project performance Moderately strong to strong positive 
relationship 
SE Capability 
Lo 
Viax. 
[to | med i [Range 
ARCH [ ) 27 29% 
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Product Architecture (ARC ) 
|< 
= Survey uestions 


This project maintains accurate and up-to-date descriptions (e.g. interface control 
documents, models, etc.) defining interfaces in detail 


Interface definition descriptions are maintained in a designated location, under 
configuration management, and accessible to all who need them 


For this project, the product high-level structure is documented, kept up to date, and 
managed under configuration control 


For this project, the product high-level structure is documented using multiple views (e.g. 


functional views, module views, etc. 


For this project, the product high-level structure is accessible to all relevant project 
personnel 


This project has defined and documented guidelines for choosing COTS product 
components 
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Moderate Higher 
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(x= 3) (36x <4) (x= 4) Gamma = 0.13 
N=17 N=11 N=18 p= 0.26 
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Configuration Management (CM) holt th 


|< ‘ 
Survey uestions 


This project has a configuration management system that charters a Change Control Board estrongly disagree 
to disposition change requests edisagree 

eagree 

estrongly agree 


This project maintains records of requested and implemented changes to configuration- estrongly disagree 
managed items edisagree 

eagree 

estrongly agree 


This project creates and manages configuration baselines (e.g., functional, allocated, estrongly disagree 
product) edisagree 


eagree 
estrongly agree 
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Best Performance 


(x > 3.0) 


Moderate 
Performance 
25<x<3.0) 


Maximum = 4.0 
3” Quartile = 3.5 
Median = 3.0 

: zs 1 ie 1° Quartile = 2.5 


Lower Performance 


a =1.0 (x < 2.5) 
Lower Moderate Higher 
Capability Capability Capability 
(xs 2.5) (2.7 <% < 3.1) (2 3.1) Gamma = 0.34 
N=15 N=16 N=15 p=0.04 


Relationship to project performance Moderately strong positive relationship 


SE Capability 
Lower Moderate PC igher 
Min. Viax. Min. Miax. Min. Max. 
frango| to | wed i |Range| [Range| to | med i [Range] [Range] to | med| i |Rance 
IPT | 34%] 4. 
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Survey uestions 


ee ree 
Proj03 This project uses integrated product teams (IPTs) 
eNo 


This project makes effective use of integrated product teams (IPTs) ehighly compliant 
elargely compliant; 
emoderately compliant 
enot compliant 


My suppliers actively participate in IPTs ehighly compliant 
elargely compliant; 


emoderately compliant 
enot compliant 


Proj07a This project has an IPT with assigned responsibility for systems engineering ehighly compliant 
elargely compliant; 
emoderately compliant 
enot compliant 

Proj07b This project has Systems Engineering representation on each IPT ehighly compliant 
elargely compliant; 
emoderately compliant 
enot compliant 
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384% 
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2 Sica - 3.0 ; i , Lower Performance 
ledian = 3.' ‘ j 
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SE Capability 
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NDIA Systems Engineering Effectiveness 38 


November 15, 2007 


NATIONAL DEFENSE INDUST: 


. rh IT tTATLAN 
P rod u ct | nte rati oO n P | STRENGTH THROUGH INDUSTRY & TECHNOLOGY 
|< 


Survey uestion 


= ee 


This project has accurate and up-to-date documents defining its product integration estrongly disagree 


process, plans, criteria, etc. throughout the life cycle edisagree 
eagree 
estrongly agree 
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Project Monitoring and Control (PMC) ee 


4.00 - 
Best Performance 
0.75 - (x > 3.0) 
Moderate 
0.50- Performance 
(225<xs3.0) 
Maximum = 3.8 
3 Quartile = 3.2 0.25 - 
Median = 2.8 Lower Performance 


1 2 3 4 1° Quartile = 2.4 
Minimum = 1.0 


(x < 2.5) 


= N= 64 0.00 - 
Lower Moderate Higher 
Capability Capability Capability 
(<= 2.5) (2.6 <x < 3) = 3) Gamma=-0.13 
N=13 N=13 N= 20 p= 0.25 


Relationship to project performance Weak negative relationship 


SE Capability 
Lower Moderate 
Min. Max. Min. Max. Min. Max. 
Jeange| to | med|_i |Range| [Range] to | wool i |Rango| [Range] to | med) i [Range 
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Project Monitoring and Control (PMC) dt 


Survey uestions (Part ) 
pio ection | Response range 


Cont13 Do you separately cost and track systems engineering activities? 
No 


Cont14a Approximately what percentage of non-recurring engineering (NRE) does systems Percentages quantized as: 
engineering represent? e<= 5% 
e<= 10% 
e<= 15% 
e<= 25% 
e> 25% 


Cont14b Is the NRE percentage estimated, or is ita measured value? eestimated 
emeasured 


PerfO1 This project creates and manages cost and schedule baselines estrongly disagree 
edisagree 
eagree 
estrongly agree 


Perf02b EVMS data are available to decision makers in a timely manner (i.e. current within 2 estrongly disagree 
weeks) edisagree 
eagree 
estrongly agree 
Perf02c The requirement to track and report EVMS data is levied upon the project6 suppliers estrongly disagree 
edisagree 
eagree 
estrongly agree 
Perf02d Variance thresholds for CPI and SPI variance are defined, documented, and used to estrongly disagree 
determine when corrective action is needed edisagree 
eagree 
estrongly agree 
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Project Monitoring and Control (PMC) hl | 
Survey uestions (Part 2) 
pio | vestion  Resonse range 


EVMS is linked to the technical effort through the WBS and the IMP/IMS estrongly disagree 
edisagree 
eagree 
estrongly agree 


OPerf05 Does this project track reports of problems from fielded items? eYes Scored by 
eNo the number 
: ; . : of positive 
OPerf06 Does the project conduct an engineering assessment of all field trouble reports? responses 
eNo 


OPerf07 The results of this engineering assessment feed into é eoperational hazard 
risk assessments 
emateriel readiness 
assessments 
esystem upgrades 
planning 
eother 
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Project Planning (PP) Vee eee 


—_ aa -—_—— N=63 


Best 
Performance 
(x > 3.0) 
Moderate 
Performance 
(2.5 <x < 3.0) 
Maximum = 4.0 
3 Quartile = 3.4 
Median = 3.0 Lower 
p oaale 7s Performance 
(x < 2.5) 
Lower Moderate Higher 
Capability Capability Capability 
s28) (28¢x<3.3) (23.3) Gamma= 0.13 
N=15 N=14 N=1? p=0.25 


Relationship to project performance Weak positive relationship 


SE Capability 
Lower 
Min. Viax. 
frange| to | wed 1 | Range 
PP 
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Survey uestions (Part ) 
pi estion | Responserange 


This project utilizes a documented set of systems engineering processes for the planning estrongly disagree 
and execution of the project edisagree 

eagree 

estrongly agree 


PDO2a This project has an accurate and up-to-date Work Breakdown Structure (WBS) that estrongly disagree 
includes task descriptions and work package descriptions edisagree 
eagree 
estrongly agree 
PDO02b This project has an accurate and up-to-date Work Breakdown Structure (WBS) that is estrongly disagree 
based upon the product structure edisagree 
eagree 
estrongly agree 


estrongly agree 


PDO2d This project has an accurate and up-to-date Work Breakdown Structure (WBS) that is estrongly disagree 
developed with the active participation of all relevant stakeholders, e.g., developers, edisagree 
maintainers, testers, inspectors, etc. eagree 


estrongly agree 


PDO3a This project6 Technical Approach (i.e. a top-level strategy and methodology to create estrongly disagree 
the initial conceptual design for product development) is complete, accurate and up-to- edisagree 
date eagree 
estrongly agree 


PDO3b This project6 Technical Approach (i.e. a top-level strategy and methodology to create estrongly disagree 
the initial conceptual design for product development) is developed with the active edisagree 
participation of those who perform the systems engineering activities eagree 


PDO2c This project has an accurate and up-to-date Work Breakdown Structure (WBS) that is estrongly disagree 
developed with the active participation of those who perform the systems engineering edisagree 
activities eagree 


estrongly agree 
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Project Planning (PP) 
* Survey uestions (Part 2) 
pio testiom | Response range 


PD04a 
PD04b 
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This project6 Technical Approach (i.e. a top-level strategy and methodology to create 
the initial conceptual design for product development) is developed with the active 
participation of all appropriate functional stakeholder 


This project has a top-level plan, such as an Integrated Master Plan (IMP), that is an 
event-driven plan (i.e., each accomplishment is tied to a key project event) 


This project has a top-level plan, such as an Integrated Master Plan (IMP), that 
documents significant accomplishments with pass/fail criteria for both business and 
technical elements of the project 


This project has a top-level plan, such as an Integrated Master Plan (IMP), that is 
consistent with the WBS 


This project has an integrated event-based schedule that is structured as a networked, 
multi-layered schedule of project tasks required to complete the work effort 


This project has an integrated event-based schedule that contains a compilation of key 
technical accomplishments (e.g., a Systems Engineering Master Schedule) 


This project has an integrated event-based schedule that references measurable criteria 
(usually contained in the Integrated Master Plan) required for successful completion of 
key technical accomplishments 


estrongly disagree 
edisagree 
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estrongly agree 
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Survey uestions (Part ) 
pio testion | Response range 


This project has an integrated event-based schedule that is consistent with the WBS estrongly disagree 
edisagree 
eagree 
estrongly agree 


PD05e This project has an integrated event-based schedule that identifies the critical path of the estrongly disagree 
program schedule edisagree 
eagree 
estrongly agree 
PDO6 This project has a plan or plans for the performance of technical reviews with defined estrongly disagree 
entry and exit criteria throughout the life cycle of the project edisagree 
eagree 
estrongly agree 


PDO7 This project has a plan or plans that include details of the management of the integrated estrongly disagree 
technical effort across the project (e.g., a Systems Engineering Management Plan or a edisagree 
Systems Engineering Plan) eagree 


estrongly agree 


PDO8 Those who perform systems engineering activities actively participate in the development estrongly disagree 
and updates of the project planning edisagree 
eagree 
estrongly agree 
PDO9 Those who perform systems engineering activities actively participate in estrongly disagree 
tracking/reporting of task progress edisagree 
eagree 
estrongly agree 
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Best Performance 
(x > 3.0) 


Moderate 
Performance 
(225<x<3.0) 


Maximum = 4.0 


3° Quartile = 3.4 44% Lower Performance 
Median = 3.0 
3 4 1 Quartile = 2.8 26% 27% (x <2.5) 
Minimum = 2.2 
———“+ -#———_ N=58 


Lower Moderate Higher 

Capahility Capability Capability 

(x = 2.8) (2.8 <x < 3.4) «23.4 Gamma = 0.33 
N= 16 N=19 N=11 z p= 0.04 


Relationship to project performance Moderately strong positive relationship 


SE Capability 
Min. 
Range 
REQ __ 33%] 4. 
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Requirements Development 


Survey uestions (Part ) 


This project maintains an up-to-date and accurate listing of all requirements specified by 
the customer, to include regulatory, statutory, and certification requirements 


This project maintains an up-to-date and accurate listing of all requirements derived from 
those specified by the customer 


This project maintains up-to-date and accurate documentation clearly reflecting the 
hierarchical allocation of both customer and derived requirements to each element 
(subsystem, component, etc.) of the system in the configuration baselines 


This project documents and maintains accurate and up-to-date descriptions of 


operational concepts and their associated scenarios 


This project documents and maintains accurate and up-to-date descriptions of use cases 
(or their equivalent) 


This project documents and maintains accurate and up-to-date descriptions of product 
installation, maintenance and support concepts 


This project has documented criteria for identifying authorized requirements providers to 
avoid requirements creep and volatility 
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Response range 
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Requirements Development Mgmt(RE ) — Synmrcnsmes 


Survey uestions (Part 2) 
pip | estion  Responserange 


This project has documented criteria (e.g., cost impact, schedule impact, authorization of estrongly disagree 
source, contract scope, requirement quality) for evaluation and acceptance of requirements edisagree 

eagree 

estrongly agree 


RDO6 The requirements for this project are approved in a formal and documented manner by estrongly disagree 
relevant stakeholders edisagree 
eagree 
estrongly agree 
RDO7 This project performs and documents requirements impact assessments for proposed estrongly disagree 
requirements changes edisagree 
eagree 
estrongly agree 


RDO8& This project develops and documents project requirements based upon stakeholder needs, estrongly disagree 
expectations, and constraints edisagree 
eagree 
estrongly agree 
RDO This project has an accurate and up-to-date requirements tracking system estrongly disagree 
edisagree 
eagree 
estrongly agree 
RD10a For this project, the requirements documents are managed under a configuration control estrongly disagree 
process edisagree 
eagree 
estrongly agree 
RD10b For this project, the requirements documents are accessible to all relevant project staff estrongly disagree 
edisagree 
eagree 
estrongly agree 
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Best Performance 


(x > 3.0) 


Moderate 
Performance 
2.5<x<3.0) 


Maximum = 4.0 
3" Quartile = 3.4 


: - Pane 28 Lower Performance 
Minimum = 2.2 i (x < 2.5) 
—_L_}+—_ N= 58 
Lower Moderate Higher 
Capability Capability Capability 
(x= 2.8) (27 <x < 3.6) (x= 3.6) Gamma = 0.28 
N=17 N=15 N=14 p= 0.061 


Relationship to project performance Moderately strong positive relationship 


SE Capability 
Moderate 
Max. 
[to | med| i [Range 
RSKM 28%] 6.1% 
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Survey uestions 


This project has a Risk Management process that creates and maintains an accurate and estrongly disagree 
up-to-date list of risks affecting the project (e.g., risks to cost, risks to schedule, risks to edisagree 
performance) eagree 

estrongly agree 


This project has a Risk Management process that creates and maintains up-to-date estrongly disagree 
documentation of risk mitigation plans and contingency plans for selected risks edisagree 

eagree 

estrongly agree 


This project has a Risk Management process that monitors and reports the status of risk estrongly disagree 
mitigation activities and resources edisagree 

eagree 

estrongly agree 


This project has a Risk Management process that assesses risk against achievement of estrongly disagree 
an event-based schedule edisagree 

eagree 

estrongly agree 


This project's Risk Management process is integrated with program decision-making estrongly disagree 
edisagree 
eagree 
estrongly agree 
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Best Performance 
(x > 3.0) 


Moderate 
Performance 
(22.5<x<3.0) 


Maximum = 4.0 
3 Quartile = 3.7 
Median = 3.0 

' . . 1* Quartile = 2.3 
Minimum = 1.0 


Lower Performance 
(x < 2.5) 


—<§$_ li i N=58 
Lower Moderate Higher 
Capability Capability Capability 
(x= 2.7) (2.7 <x 3.3) (x= 3.3) Gamma = 0.37 
N=18 N=12 N=16 p=0.03 


Relationship to project performance Moderately strong to strong positive 
relationship 
SE Capability 
Lower Moderate TT gher Cid 
Min. Viax. Min. Max. Min. Max. 
Jeange| to | mea] i [Range] [Range] to | med 1 [Range] [Range| to | moo] 1 [Range 
TRADE | 3.3 | 19%] 32%] 49%] 4.0_| 
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Trade Studies (TRADE prc hnaiaimcenitisys 
Survey uestions 


Stakeholders impacted by trade studies are involved in the development and performance estrongly disagree 
of those trade studies edisagree 

eagree 

estrongly agree 


This project performs and documents trade studies between alternate solutions based upon estrongly disagree 
definitive and documented selection criteria edisagree 

eagree 

estrongly agree 


Documentation of trade studies is maintained in a defined repository and is accessible to all estrongly disagree 
relevant project staff edisagree 

eagree 

estrongly agree 
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; , Be a 
Technical Solution (TS ttl 1k 


Maximum = 4.0 
3” Quartile = 3.3 
- : Median = 2.9 

; : _ 1* Quartile = 2.6 
Minimum = 2.1 


= N=57 


Note: TS is a composite measure equivalent to ARCH + TRADE. 


Best Performance 
(x > 3.0) 


Moderate 
Performance 
(2.5<xs3.0) 


Lower Performance 


(x < 2.5) 
Lower Moderate Higher 
Capability Capability Capability 
(x= 2.8) (2.8 <x < 3.2) (X= 3.2) Gamma = 0.36 
N=15 N=15 N=15 p= 0.03 


Relationship to project performance Moderately strong positive relationship 


SE Capability 


TS 
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Technical Solution (TS) 


Survey uestions (Part ) 


Stakeholders impacted by trade studies are involved in the development and 
performance of those trade studies 


This project performs and documents trade studies between alternate solutions based 
upon definitive and documented selection criteria 


Documentation of trade studies is maintained in a defined repository and is accessible to 


all relevant project staff 


This project maintains accurate and up-to-date descriptions (e.g. interface control 
documents, models, etc.) defining interfaces in detail 


Interface definition descriptions are maintained in a designated location, under 
configuration management, and accessible to all who need them 
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Technical Solution (TS) 
Survey uestions (Part 2) 


For this project, the product high-level structure is documented, kept up to date, and 
managed under configuration control 


For this project, the product high-level structure is documented using multiple views (e.g. 
functional views, module views, etc.) 


For this project, the product high-level structure is accessible to all relevant project 
personnel 


This project has defined and documented guidelines for choosing COTS product 
components 
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Best Performance 


(x > 3.0) 
Moderate 
Performance 
(2.5<xs 3.0) 
Le = et Lower Performance 
uartile = 3. i 
Median = 3.0 i (x < 2.5) 


; 2 - 4 1° Quartile = 2.7 
Minimum = 1.7 


EE N=58 


Moderate Higher 
Capability Capability Capability 
(Xs 2.7) (2.7 <x < 3.3) (= 3.3) Gamma = 0.28 
N=13 N=12 N= 21 p=0.07 


Relationship to project performance Moderately strong positive relationship 


SE Capability 
VAL 
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alidation (AL) hl ld th 
Survey uestions 


V&V04a This project has accurate and up-to-date documents defining the procedures used for the estrongly disagree 
validation of systems and system elements edisagree 
eagree 
estrongly agree 


V&V04b This project has accurate and up-to-date documents defining acceptance criteria used for estrongly disagree 
the validation of systems and system elements edisagree 
eagree 
estrongly agree 


This project maintains a listing of items managed under configuration control estrongly disagree 
edisagree 
eagree 
estrongly agree 
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Best Performance 


(x > 3.0) 
Moderate 
Performance 
22.5<xs3.0) 
Maximum = 4.0 . r r Lower Performance 
3" Quartile = 3.4 ‘i ‘ ’ 
Median = 3.0 i (x < 2.8) 
1* Quartile = 2.6 
Minimum = 2.2 
— = ==}_—_ N=58 


Lower Moderate Higher 

Capability Capability Capability 

(= 2.7) (2.7¢x<3.2) (23.2) Gamma = 0.25 
N=16 N=15 N=15 p=0.09 


Relationship to project performance Moderately strong positive relationship 


SE Capability 
Min. A A 
range] Lo | we [a we] Lo | wed 
VER 25%] 9.0% 33% 33%] 20%] 47%] 4.0 _| 
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* Survey uestions (Part ) 


V&VO01a This project has accurate and up-to-date documents defining the procedures used for the estrongly disagree 
test and verification of systems and system elements edisagree 
eagree 
estrongly agree 


V&VO01b This project has accurate and up-to-date documents defining acceptance criteria used for estrongly disagree 
the verification of systems and system elements edisagree 
eagree 
estrongly agree 
V&V02a This project has a documented and practiced review (e.g. peer reviews, design reviews, estrongly disagree 
etc.) process that defines entry and exit criteria for work products edisagree 
eagree 
estrongly agree 


V&V02b This project has a documented and practiced review (e.g. peer reviews, design reviews, estrongly disagree 
etc.) process that includes training requirements for the reviewers edisagree 
eagree 
estrongly agree 
V&V02e This project has a documented and practiced review (e.g. peer reviews, design reviews, estrongly disagree 
etc.) process that addresses identified risks and risk mitigation activities during reviews edisagree 
eagree 
estrongly agree 


V&VO02F This project has a documented and practiced review (e.g. peer reviews, design reviews, estrongly disagree 
etc.) process that examines completeness of configuration baselines edisagree 
eagree 
estrongly agree 
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. Survey uestions (Part 2) 


This project conducts non-advocate reviews (e.g. reviews by qualified personnel with no estrongly disagree 
connection to or stake in the project) and documents results, issues, action items, risks, edisagree 
and risk mitigations eagree 

estrongly agree 


V&V02c This project has a documented and practiced review (e.g. peer reviews, design reviews, estrongly disagree 
etc.) process that defines criteria for the selection of work products (e.g., requirements edisagree 
documents, test plans, system design documents, etc.) for review eagree 

estrongly agree 


V&V02d This project has a documented and practiced review (e.g. peer reviews, design reviews, estrongly disagree 
etc.) process that tracks action items to closure edisagree 
eagree 
estrongly agree 
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Lower Performance 
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Lower Moderate Best 
Capability Capability Capability 
s2.8)  (28<x<3.1)) 23.1) Gamma = 0.49 
N=15 N=13 N=18 p= 0.005 
Relationship to project performance Strong positive relationship 
SE Capability 
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(x $2.5) (2.5<x<3.0) (x 23.0) Gamma = 0.32 
N=13 N= 1? N= 16 p=0.04 


Relationship to project performance Moderately strong positive relationship 


SE Capability 
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Project challenge factors 
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Technical complexity on ee vee 
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Driving Factor Relationship to Project Driving Factor Relationship to Project 
Performance Performance 


Requirements and ery strong Total Systems Moderately strong 
Technical positive Engineering Capability | positive 
Se Project Challenge Moderately strong 
wi rojec allenge negative 
alidation Moderately strong 
positive 
Risk Management Moderately strong 
positive 
erification Moderately strong 
positive 


Product Integration Weak positive 
Project Planning Weak positive 


Configuration Weak positive 
Management 

Project Monitoring and | Weak negative 
Control 


Process Improvement | Weak positive 
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A Mission 


| Ensuring Information Superiority and Agile 
Combat Support; Providing a Wide Range of 
Acquisition and Sustainment Services to the 
Warfighter — Through Teamwork, Innovation 
and Technological Excellence 


A Organization 
| 800+ personnel 
| Lackland AFB (San Antonio), Texas 
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A Programs 
| Air Force Public Key Infrastructure 
| Air Force Common Access Card (CAC) 


| Air Force cryptologic equipment depot and 
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| Air Force Cryptographic Modernization 
Program Office 
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Category ESC Process Areas 


Project 
Management 
OD 
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Risk Management Technical Project 
Planning 


Support : 
eee Configuration Management uality Assurance 


Life-Cycle Logistics 


engines ing Requirements Dev Mngt Enterprise Integration 


Integrated Testing System Safety 


Process 
Management No process areas identified 
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: Red text 1 New ESC process areas 
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Training 
Links 
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EN Process Improvement 


Welcome to the EN Process Improvement Resource Center 


Since 1998, a government-industry-Software Engineering Institute (SEI) collaboration has been under way to 
develop a product suite of models, training, and assessment methodology that support integrated process 
and product improvement across the enterprise. These products are intended to replace legacy maturity 
models,including SW-CMM and Electronics Industries Association Interim Standard (EI4/1S) 731, the Systems 
Engineering Capability Model (SECM) in December 2003. 


Toolkits 


Configuration Management Toolkit * 
Enterprise Integration Toolkit 
Integrated Testing Toolkit + 
Life-Cycle Logistics Toolkit 
Partnering Toolkit 

Quality Assurance Toolkit 
Requirements Process Toolkit* 

Risk Management Toolkit + 

System Safety Process Toolkit 
Technical Project Planning Toolkit 


* CPSG Focus Areas 
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ESC Toolkit 
Step 1 - CM Process 


Preparation 
and 


1 
Hierarchy 


Action Appoint an Enterprise Configuration Manager and Component System 
CMs and Develop Implementation Strategy 


A Configuration Manager needs to be appointed for the program as well as a 
support team to handle the Integrated Digital Environment and any automated 


configuration management tools to be used on the program. An 
implementation strategy needs to be developed that addresses the 
requirements for the configuration management effort . 


ESC Major Step 
(required) 


Sub Steps 


Strategy 

ierarchy ESC Actions 
Control (Optional) 
Stakeholders 


Buy-in 
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a... Tailoring Guidance 


Required 


The major steps are the goals of each process. All organizations are required 
to implement each process that achieves these goals. 


Optional 


The actions (e.g., 1a, 1b, etc) for each step are considered best practices and 
are expected to be performed by each organization to implement satisfactory 


processes. It is possible to satisfy the required goals without implementing the 
expected practices but the burden of proof is on the organization using an 
alternative set of practices. 


Suggested 


All material covered in the training sessions and resources provided in the 
toolkit are suggested approaches to implementing the expected practices. This 
material is optional and may be used at the discretion of the organization. 
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ASix Process Areas for Progzam Implementation 


| Life-Cycle Logistics 
1 Technical Project Planning 
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rocess Guide Example 


3.2. Configuration Management Practices 

Using the ESC provided guidance, CPSG/EN has developed a Configuration Management 
Implementation Guide which provides additional guidance when developing a program specific 
CM process. The ten areas described below must be implemented. 


ed Features 


3.2.1. Preparation and Hierarchy 
In this area, each acquisition program, system, and end item is required to: 


Appoint, in writing, a configuration manager 
Develop a CM plan 
Identify the configuration items (CI’s) 


Identify the internal and external stakeholders 


Determine the CM structure and hierarchy 


Establish a Configuration Control Board (CCB) 


ESC Required Steps 


Identity tool requirements 


CPSG Required Steps 


Coordinate tool requirements with CPSG/EN 


Train their CM workforce 


Update CM information on ENWeb 
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Implementation Guides 


A |lmplementation Guides 
1 Contain the fHowo 
| Allowable program tailoring identified 
| Templates provided for each process area 


| Provide Program Managers/Lead Engineers 
with an r80%osolution 


| Ensure consistency across CPSG 
| Example: Configuration Management Process 
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— em Plan Development and 
—— Tailoring Guidance 


4.1.4 Determine Configuration Management Structure and Hierarchy 


A government configuration management program needs to be established for each program to 
handle the user requirements, system requirements, and external interfaces. For each 
development contract awarded under the program, the contractor will probably be required to 
establish a configuration management program to handle the system requirements, allocated 
requirements, product requirements, and support requirements. Once the system is fielded, a 
sustainment configuration management program needs to be established. In a system of systems 
program or one involving multiple development contracts, a configuration management hierarchy 
must be established. 


Implementation: 
CM Plan Template: Configuration Management Organization 


Tailoring Guidance: 


This is always required, but the actual structure is left to the program. 
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ESC/EN Web 


CPSG CMMI Responses 

Requirements Development and Management Level 2: Complete Status: (B) Updated: 30 Nov.06 
Undated: EG 
Undated: 30 Nov. 06. 
Undated: iG 


Undated: i 


Configuration Management Level 2: Complete Status: (& 
Risk Management Level 2: Complete Status: (B) 


Enterprise Integration Level 2: 11 Mar. OF Status: (@ 


Integrated Testing Level2: 04 Feb. 0% Status: 


”) 
© 
v 
= 
”) 
“) 
®o 
O 
2 
Ge 
= 
= 
O 
O 
VY) 
LL 


Technical Project Planning Level 2: Complete Status: (8) Updated: 30 Nov. 06° 
Quality Assurance Process Level 2:04 Feb. 07 Status: Undated: 
aystem oafety Process Level 2: OF Jan. OF status: Undated: 
Life-Cycle Logistics Level 2: Complete Status: (8) Undated: EG 
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Generic Goals 


Part Il: Generic Practices 


= : Perform Development Base Practices Lev 2 © Yes 

e GP 1.2: Perform Management Base Practices Levi 2 ®@ Yes 

O 

Y) 

LL 
GP 2.1.1: Establish an Organizational Policy Lev: 2 No 23 Feb, O7 
GP 2.1.2: Plan the Process Lev 2 © Yes 
GP 2.1.3: Provide Resources Lev 2 © Yes 
GP 2.1.4: Assign Responsibility Levi'2 ®@ Yes 
GP 2.1.5: Train People Lev. 2 © Yes 
GP 2.1.6: Manage Configurations Lev: 2 No 268 Feb. 07 
GP 2.1.7: Identify and Involve Relevant Stakeholders Levi 2 ™ Partial 31 Jan. O7 
GP 2.1.8: Monitor and Control the Process Lev 2 ©@ Yes 
GP 2.1.9: Objectively Evaluate Adherence Lev; 2 No 01 Jun. OF 
GP 2.1.10: Review Status with Higher-Level Management Lev: 2 No 01 Mar. OF 
GP 2.1.11: Perform Base Practices Ley. 2 © No 01 Jun. 0? 
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SC CMMI (ENWeb) 
rements Process Specific Goals 


Requirements Development and Management Level 2: 01 Jun. OF status: (s) 
Question Answer Planned Date Status 
a Part |: Specific Practices 
5 SG 1: Develop Customer Requirements 
O SP 1.1: Collect Stakeholder Needs Lev. 2 © Yes (s) Edit 
= 
‘oO SP 1.2: Transform Stakeholder Needs, Expectations, Levi:2 ® Yes (@) Edit 
fou Constraints, and Interfaces into Customer Requirements 
Y) “ ; ' 
= SP 1.3: Elicit Needs Lev:'2 @ Yes (@) Edit 
S $G 2: Develop Product Requirements 
O SP 2.1: Establish Product and Product Component Levi 2 © Yes (s) Edit 
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Presentation Objectives 


Describe lean principles. 


Discuss how to define lean (e.g., short and 
usable) processes and procedures. 


Present some lean metrics. 


Describe some lean success stories (including 
service industry). 
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Some Lean Success Stories 


Summary 
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he Quality Crisis 


The cost of poor quality 


e “In most companies the costs of poor quality 
runat2 to percent... In other words, 
about 2 to percent of the companies’ 
efforts are spent in redoing things that went 
wrong because of poor quality” (Juran on 
Planning for Quality, , pg. ) 


e Crosby’s’ uality Management Maturity Grid 
states that if an organization doesn’t know 
it’s cost of quality, it’s probably at least 2 
(Crosby, Quality is Free, ,pg. - ) 
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Lean Problems 


Most organizations have too much waste (e.g., 
non-value added). 


Most processes have too many “non-value 
added” steps. 


Ow can organizations focus on “value added” 
and remove waste 


Lean is a recent quality approach to help 
organizations focus on “value” and remove 
“non-value”. 
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What is Lean? 


Lean has its roots in quality and manufacturing, 
and is a recent popular movement in quality. 


“Lean Production” is the name for the Toyota 
Lean Production System. 


The following are major lean references (see 
references in back of presentation for full 
references) 


rn he Machine That Changed The Worldo 
rLearning to Seeo 

nmhe Toyota Wayo 

rn he Toyota Product Development Systemo 
fLean Thinkingo 
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Quality Maturity 


“We know why we have 


Prevenor happy customers.” 


“uality planning, control, and 


Wellness ; Be S 
improvement are routine. 


“Management commitment and 
continuous improvement 
resolve quality problems.” 


Progressive 
Care 


Intensive “We don’t know why we have 
Care quality problems, but they hurt.” 


Comatose __| “What quality problems ” fe |] 


e Acronyms are (CO Costof uality BA Baldrige Award DCF Dilbert Correlation Factor SEI SEI CMMI CMM) 


¢ Based on “The Eternally Successful Organization”, by Crosby, the SEI, the Baldrige Award, Dilbert Comics 
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e Lean Principles - (1) 


Establish customer defined value (i.e., identify 
the “value stream”). Process “value”. 


Continuously eliminate non-value added 
activities (e.g., waste, rework, defects). 


Use leadership and standardization to create a 
lean culture. 


Align your organization through visual 
communication. 


Create an optimized process flow (e.g., “Flow”, 
“Pull”, “Just-In-Time”, “Leveled”’). 
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Lean Principles - (2) 


Use lean metrics to manage the value stream. 


Front-Load the process for maximum design 
space. 


Build a learning organization to achieve lean 
and continuous improvement. 


Adapt technology to fit your people and 
processes. 


Strive for perfection through continuous 
improvement. 
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vymmon Process Problems 
Too Big Processes become too large and complex 


Poor Usability Not “fit for use” by process users. 
any processes contain mixed information types 


Poor Design Process documentation usually 
violates good definition and writing principles 


Not _isual Processes need to be defined by well 
thought out diagrams, pictures, or models 


Defined Sequentially Processes are not novels 


Shelfware or Unused Webware Unused processes 
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Why Lean Processes? 


Reduce Size Lean processes are precise, concise, 
and more usable. 


Better Usability Lean tele telat require defining 
chunks” and labeling them for use. 


Better Designs Lean processes require good 
process definition and writing principles. 


isual Diagrams Processes need to be defined by 
well thought out diagrams or “process models”. 


Defined Non-Linearly Processes are dynamic, 
parallel, and concurrent. 
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uldelines for Lean Processes 


Chunk steps (_ plus or minus 2) into usage 
scenarios (e.g., plan, control, improve, engineer). 


Use process modeling and best practices (e.g., 
procedures, standards) to select the best chunks. 


uestion every step of the process. 
Remove “non-value added” steps. 
Combine similar steps. 


Refine steps to be short and usable. 
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POLICIES ) STANDARDS 
“Laws” or “principles” . “Operational definitions’ 
that govern operations ey one for work products 


PROCESSES 
“What happens over 
time” to build products 


Implemented by 


PROCEDURES 


“ow to” or step by 
step instructions 


| 
Supported by 
TRAINING TOOLS 
Provides the needed | <«———————————— | Sup ports and 
automates operations 


knowledge and skills 


ASlide adapted from0\ Software Process Framework for the SEI Capability Maturity Modelg Olson, Timothy G., et al, CMU/SEI-94-HB-01 
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ey Process Questions 


Who does what activity? 2. Role(s) 
What work products are used? 3. Input(s) 


What work products are produced? | 4. Output(s) 


When does the activity begin? 
When does the activity end? 
Where is activity performed? 
What activities are performed? 
How is the activity implemented? 


ASlide adapted froma Software Process Framework for the SEI Capability Maturity Model Olson, Timothy G., et al, CMU/SEI-94-HB-01 
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Beginner Mode 
Assumes user has little or no experience 


Includes training material 
Includes process guidance and lessons learned 


Intermediate Mode 
Assumes user has some experience with process 


Includes process guidance and lessons learned 


Expert Mode 


Assumes user is very experienced 
Concise and precise as possible 


All Modes should be “Chunked” and Include 
Process models, checklists, forms, and tables 


Pointers to detailed explanations 
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e Stream Mapping 


Define the product service process to map. 
Define the current value stream map. 
Define the future value stream map 


Implement the work plan (i.e., future state) 


Manual Flow 
Information 


Process Box 


(Example 
Notations) 


¢ Adapted from Keyte, Beau, and Locher, Drew. The Complete Lean Enterprise: Value Stream Mapping for Administrative and 
Office Processes, Productivity Press, New ork,N ,2. 
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Example Lean Metrics 


Takt Time Data Box 
Lead Time 
Process Time 
Changeover Time 
Available Time 
alue-Added Time 
Demand Rate 
Number of People 
Inventory 
Percent Complete and Accurate 
Information Technology Used 
Reliability 
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Lean Measurement Framework™ 


GOALS KE  UESTIONS METRICS 


Cost, defects, 
PLAN effort, size, 
schedule, etc. 


Cost, defects, 
IMPRO E effort, size, 
schedule, etc. 


Cost, defects, 
CONTROL effort, size, 
schedule, etc. 


*DC DataCollection DS Data Storage 
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e Lean Success Stories 


Toyota - Best lean documented success 
story in industry (from manufacturing - 
see references). 


ewlett Packard -a CMMI success story 
of a software service. 


NASA Decision Analysis Process - A 
lean decision analysis process that 
scales up to complexity. 
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Lean CMMI Process 


The Capability Maturity Model Integrated 
(CMMI ) is an industry systems engineering 
model used in engineering, service, software, 
automotive, etc. 


A small unit of P implemented CMMI Maturity 
Level inabout months (anaverage year 
effort). 


The lean P Maturity Level process is about 
2 ofthe size of the P India Process (ora 
typical CMMI Maturity Level process ). 


¢ Olson, Timothy G., Kellum, Julie, and Tufail, ia., “Rapidly Defining a Lean CMMI Maturity 
Level Process”, Presentation, NDIA CMMI Conference, 2 
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ASA Lean Process 


Used best practices to define a lean decision 
analysis process model (actual process model 
IS pages-in_ page “chunks’”). 


Developed a website to help automate the 
process. Everything is “one click away”. 


Process works for simple and average 
decisions, and also scales up to complex 
decisions. 


The process (complete with policy, standards, 
processes, and procedures) is about pages. 
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xample Decision Analysis Process 
Model 


Prepare 2 Conduct . Perform 


for Decision [| | Decision 
Decision Meeting Follow-up 


¢ Olson, Timothy G., “Using Best Practices to Define a Lean Decision Analysis Process”, Presentation, 
INCOSE Systems Engineering Conference, 2 
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2 Conduct Meeting 


Input Entry Criteria Purpose 
Roles Activities 
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Decision Team Decision 
Recorder 


Representative Maker(s) 
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6.2.1 
Present Decision 


Package to 
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Summary 


Lean Organizations 
Define customer value (i.e., the “value stream’’) 


Process “ alue Added” 

Remove non-value added (i.e., waste) 
isualize the process 

Optimize process flow 

Manage the process with lean metrics 

Create and sustain a lean culture 

Build a lean learning organization 

Adapt technology to fit people and processes 

Strive for perfection through continuous 

improvement 


A 
A 
A 
A 
A 
A 
A 
A 
A 
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Almost all acquisition programs get into cost and schedule trouble at some 
time during the program. 


A few months before a program breach, all the indicators are Green. 


One month later, we receive notice that the program is late -- Red -- and 
will have to re-baseline. 


What indicators can we use that warn us of the upcoming problems? 


Is there time for us to take action to improve our likelihood of Success? 


Mr. Durante (SAF/SAQ) asked us to investigate the possibilities. 
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Presenting information about program problems & risks that can be 


provided in time to take some corrective action. 
A Earned value can be used to estimate completion time and cost 


Existing examples: 
A Process performance predicts team results 


Conversely, status reports, and most progress indicators reflect past 


A Testing and peer review results predict product quality 
performance unless the representation of the data helps us to predict 
y\ a m--Celbalem lave l (orice) a) 


Bob Ferguson, Nov, 2007 
© 2007 Carnegie Mellon University 


project outcomes. 
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Our work on program leading indicators has been grounded in: 
A Patterns of program success (published literature) 

A Patterns of program failure (Aerospace Corp and SEI work) 


A Analysis of measurement use within successful programs 
A Lean Aerospace Initiative: joint work of MIT, INCOSE and others 
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cause; however, we can identify some distinct patterns that allow problems 


Program failure usually cannot be tracked to single risk, event or single 


to fester. 
Many problems seem to be rooted in ordinary sources of uncertainty that 


are natural to the various stages of product development. 
Failure to deal with the uncertainty in a constructive manner causes a 
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Sources of Uncertainty and Potential for Action 


Representation 


Method 
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We must learn some new capabilities and develop new capacities in order 
lf there was nothing to learn, we would not call ita development program. 


to develop a new product. 


No risk = No reward 
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What is it that we do not know when we start? 
How can we observe that we are learning these things? 
How can we observe that we are developing the needed capability and 


Capacity for the work? 
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Who can we influence if the right things are not happening? 
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Status 
Exceptions 
Risks & Issues 
Change requests 


Evaluation results 
Priorities 
Requirement changes 


Mission changes 
Funding changes 
Resources 

Schedule targets  meseeerrereeeeee fesse Gevssssscrcsceseeeeeeeeeeeenenennnennssnnyy at ieceeeeeeeeraaa TTT 
Policy directives 


Sponsor 


A Process performance 
effectiveness 

A Quality of internal 
products & services 


MY 
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Intermediate products 
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Status and Progress 
Technical Performance 
Risks 

Change requests 


Feedback 
Change requests 
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q, Program Office =| | Estimation and Planning 
Performance | Assumptions 


Identifiable sources 
of uncertainty 


@ ‘Stakeholder | 
_ Influence 


@} Contractor Performance / 


| Gentract and Relationship | 


: 4 Technology and 
_ Integration 
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ning Uncertainty 


Feasibility 
A Types of users 
A External interfaces 
A Constraints 
ConOps 
A Feasible performance bounds 
A Understanding of user need 
Specifications 
A Secondary functions 
A Storage needs 
A Optimal architecture 
A Developer skills 


A Reliability 
A Achievable performance 
A Tester understanding of scenarios 
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¥ Complete 
A Lack of transparency and trust (calls for occasional review of relationship) 


feed features 


values and work toward common goals. 
Disconnects between program office and contractor 
A Increasing numbers of action items and issues. 


Possible drivers 
Bob Ferguson, Nov, 2007 
© 2007 Carnegie Mellon University 


A Too few or too many communications links. 
6 Too few means that communications take too long. 
A Personnel turnover or role changes on either side. 


6 TOO many means that decisions can become invisible. 
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Technology Readiness Level (TRL) is evidence of the maturity of the 
Design rules for using the technology including interfaces, timing, and other 


other fronts as well. 
technology (but not the process and other elements) 
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A 
aspects of system architecture 


How to verify and validate (test) components with the new technology 


A 
How to write documentation for users and support 


To 


Tooling and manufacturing 
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ae features 

Product integration results provide the best evidence of progress. 
A Project management requires multiple integration points 

A Systems engineering and architecture identify integration points 
A Verification activities demonstrate low level functionality 

A Validation activities demonstrate fitness for usedfor other development 
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Staff. 
A Requirements change by 2% per month 
A We need to minimize the rework that results from these changes; therefore 


Bob Ferguson, Nov, 2007 
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Failure to integrate often allows for some drift phenomena that are difficult 


to manage. 
A Affects design and test at 3x-5x rate. 


we must discover the effects sooner. 
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feed Features 


Required Learning: Are any teams underperforming (vs. plan)? 
Resources 

A Availability, capability and utilization compared to plan 
Process 

A Process assurance and rework measures 

A Process performance by team (fsay-dooratio) 
Earned Value Management System (EVMS) 

A Compares cost and task completion to plan 
Problem Drivers: 

A Extreme schedule pressure 

A Internal process problems (Including hiring practice) 


“Projects fail one team at a time” Bob Ferguson 
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Status 
Exceptions 
Risks & Issues 
Change requests 


Evaluation results 
Priorities 
Requirement changes 


Mission changes 
Funding changes 
Resources 

Schedule targets  meseeerrereeeeee freee Gevssssssssreeseseeeeeeeeeeeenenentntsennny 
Policy directives 


Sponsor 


AProcess performance! Integrato 
effectiveness 

A Quality of internal 
products & services 


Suppliers 


Intermediate products 
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Status and Progress 
Technical Performance 
Risks 

Change requests 


Feedback 
Change requests 
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All the usual problems exist. 
A Resources 
A Process 
A Quality Assurance and Rework 
A Schedule 
A Change Requests and Volatility 
There is rarely any measurement data that addresses internal 
performance. 
There is rarely a real project plan in use within the program office. 
Only a few processes are documented and trained (Source selection is). 
AF Leading Indicators 
con) cuteness ihe 
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6 Volatility in either direction causes some change impact. 
6 Lack of participation by users and sponsors as agreed is a strong 


3 tL 


Stakeholder Influence 


A Funding 
A Specification and Validation 
6 Both must participate in validation work. 
6 Executives must be responsive to change requests (yes/no) 


warning 
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A Sponsorship 
6 Advertising 


6 Interest and support for the work 
A Timely participation, too much senior officer interference 


User Community Problems 
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Technology 
alternative sources 
capability development 


Contractor Relationship 

commitment and trust 
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External Stakeholders 
commitment and trust 

Program Management (synchronization) 

verification validation results and change management 


team readiness, product readiness 
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7 ame features 
Sources of Uncertainty and Potential for Action 


Representation 


Method 
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feemieed Features 


Different stakeholders need different representations of the data. 

A Stoplights might be ok for executive communications 

A They are not sufficient for PM and program office staff. 
The representation should communicate the opportunity for action. 

A Who needs to know? Who can we influence? How urgent is the action? 
Learn to ask the right questions. 


A When you select the indicator (charté ), think about what you might see if 
something bad were happening? How would you ask questions to confirm 
your judgment? 


A What does it mean if the graph goes down instead of up? If a threshold is 
crossed? 
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Systems Engineer 
Contractor Project Management 


Program Management 


PEO 
Acquisition Executive 
rSchoolhouseo 
Field Commander 
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Tae Features 


Continues to be a problem. 

Does capability exist (implemented)? 

Examine schedule for integration points. 

What verification and validation tests represent fyood enough? 
What organizational capability is needed for use at this point? 
Confirm/agree with Contractor to these quality criteria. 


Create Kiviat diagram showing the various dimensions of this TPM with 
TPI, MOE, MOP measures represented 


Review after each integration. 


Progress is evidenced when all dimensions are at the desired level. 
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NOTES: 
Inside Red circle is unacceptable risk. 


Inside Yellow circle is manageable risk 
Outside Yellow circle is “Green” 


Verification Capability 


Multiple axes represent 
the different things that 
have to be learned to 
deploy the technology 


Rings can represent 
whether the level of 


ig completion is 
acceptable. 
AXIS4 Interface 
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B Actual 137 126 118 124 116 120 125 
0 Delta 27 22 45 29 46 26 37 
O Delta% 16% 15% 27% 19% 28% 18% 23% 
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2sentation 
: 230 person months behind 


[555 Forecast 
GS Actual 
—— Cum. 


Dec- Jan-07 Feb-07 Mar-07 Apr-07 May-07 Jun-07 Jul-07 


06 
Dec-06 Jan-07| Feb-07)| Mar-07 Apr-07 May-07 Jun-07 Jul-07 
Forecast 164 148 162 152 162 146 162 150 
Actual 137 126 118 124 116 120 125 


(26) (37) 
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Sources of Uncertainty and Potential for Action 


Representation 
Method 
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feed Features 


Diagnostic (based on Alberts & Dorofee) 
A ldentify principle drivers and nature of uncertainty 
A Prioritize near term concerns 
Goal-driven measurement 
A Establish goals for reducing uncertainty 
A Ask questions about how to understand it with measurement 
A Ask questions about success criteria for management 
A Create useful indicators (charts) 
A Plan for data collection and implementation 
A Assign individual responsibility 
Contractor 
A Plan the contractor questions 
A lIdentify (with contractor) how questions will be addressed 
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Diagnostic is an interview. 
A Each participant requires about 30 minutes. 
Goal-Driven Measurement Is a facilitated method 


A Can be effectively performed in 2 days or better in a sequence of 5-6 
2-hour meetings that can be accomplished in a week. 


A PM, deputy and chief engineer should attend the first meeting. 
A Method is followed by a briefing to PM. 


Working with the contractor requires approximately 2-3 days at contractor 
site. It includes face to face meetings with program management and IPT 
leads. 
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Lean Aerospace Initiative: several interesting publications 
http://lean.mit.edu 

Aerospace Corp: rBell-ringing Criteriaoand rPatterns of Program Failureo 
Suellen Estlinger, Richard Ableson, available at SSTC programs 

Fred SchenkerG presentation at this conference 
rProject Management by Functional Capabilityo 

Army and Air Force fProbability of Program Successo 


See DAU website and IT-COP 
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First pilots are just starting | 1 in the Army in 1 in Air Force 
The diagnostic does help us narrow the concerns quickly. 
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Overview 


AGeneral Motors now acquires of its IT solutions 


AThere was no maturity model for organizations that 
rYore (UII c=Yomc-Leq alate) rere hv, 


AGeneral Motors supported the Software Engineering 
Institute (SEI) to develop the CMMI-AC 


AGeneral Motors was the first commercial enterprise to 
be appraised utilizing the CMMI-AC 


AThe best practices encompassed in the CMMI-AC 
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GM’s Profile 


A 280,000 employees worldwide 

A $207 billion revenue in 2006 
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A 14,000+ dealers in North America alone 

A $89 billion of direct materials purchased annually 
A 14 million pounds of material received daily 

A Approximately 5,000 parts in each vehicle 
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AStrategic decision to 
acquire - not build 


APreviously utilized 
CMMI development 


ARecognition that 
Acquisition is 
different 

ARequires standard 
model for global 
deployment 
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AThought Leadership 
in developing 
maturity models 
(CMM, CMMI) 


Alntegrator of best 
practices in software 
engineering 

ARecognition of value 
in offering model for 
acquirers 


AS eeeetAcquisition (CMMI-AC ) ; 


A Maturity Model for organizations that acquire systems 
ALeverages best practices of industry and CMMI 


A Publication November 2 
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Key Learnings 
Define the process for the organization and interface 


to suppliers 


Focus delivery process on acquirer core 
competencies 


Doné prescribe the suppliers methodology 


Supplier teams integrate at different speeds and 
require a variety of support mechanisms 
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Key Learnings 


& Acquirer ownership of requirements is essential 
Intellectual property, business knowledge 
Relationship with customer/user 

Continuity between projects 


Requirements 


Architecture 


Project Management 


System Delivery Process 
Contract Management 
Integrated Tools 


Core 
Competencies 


= Requirements are tightly linked to contracts 
Tension within teams if reqts. are poor quality 
Suppliers can take advantage of loose reqts. 


© Acquirer must be skilled in requirements 
management and requirements engineering 


ARequirements prototyping 
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Key Learnings 


& Acquirer must retain ownership of architecture 
Determines technical strategy 
Requirements Assure consistency 
Architecture) Assure @®est Interest6of Acquirer 


_- Propeerivverrergerrient 


—— E Architectural philosophy varies within supplier 


base 
= 
—— Variation at company and personal level 


E Technical and architectural standards eliminate 
noise 
Hurdle should be very high for deviations 
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Key Learnings 
E Acquisition PM is a different mindset than 
development 
PMG shift from rmoingoto fmanaginga retain 
accountability 
: Acquisition PM must integrate with supplier PM 


Clear acceptance and quality measures are critical 
E Relationship with customer its critical 
PM is the key customer advocate in the project 


& Contract support of PM is critical 
Contracts routinely referenced for responsibilities 


Core = 
Competencies y Architecture 


. Project Management 
ee” il 


Integrated Tools 
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ACMMI-AC focuses on Acquisition Maturity 


AAcquirer Must Retain Key Competencies 
ARequirements 
AArchitecture 
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AAcquirer Must Drive uality Processes 
ARequire suppliers to conform 
ALeverage learnings from suppliers 
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Central themes 
A Baselines 
A Control Charts 


A Statistical management 
of subprocesses 
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Central themes 


A Process Performance 
Models 


A Understanding and use 
of variation 


Supporting themes 
A Baselines 
A Control Charts 


A Statistical management 
of subprocesses 
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s Presentation 


A “BAD” INTERPRETATION 
Text in yellow boxes is an example description of implementing the 
practice consistent with the glossary, using the standard English 
meaning of words instead of the statistical meaning, and without 
using the informative material. For example, interpreting variation to 
mean the difference between two items. 


A “GOOD” INTERPRETATION 
Text in green boxes is an example description of implementing the 
practice consistent with the glossary, the statistical meaning of 
words, and accounting for the informative material. For example, 
interpreting variation (in the level 4 & 5 practices) to mean central 


tendency and dispersion. 


Selected practices are illustrated. 
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The MDD states on page I-20 


A "Appraisal teams compare the objective evidence collected 
against the corresponding practices in the appraisal reference 
model. In making inferences about the extent to which practices 
are or are not implemented, appraisal teams draw on the entire 
model document to understand the intent of the model, and use 
it as the basis for their decisions. This comparison includes the 
required and expected model components (i.e., generic and 
specific goals, generic and specific practices) as well as 
informative material, such as model front matter, introductory 
text, glossary definitions, and subpractices." 


igh Maturity ow Do We Know 
SAUL Amel Lalo pm =o) ome) exe (or-lee mm IL Com aColale-le| 


=> Software Engineering Institute | CarnegieMellon November 2 


2 Carnegie Mellon University 


Your complimentary 
use period has ended. 
Thank you for using 


PDF Complete. Informative Material and 


Click Here to upgrade ft 
Unlimited Pages and EXpiau-—=——a 


Additionally on page I-24 in discussing direct artifacts for PIls 


A "The tangible outputs resulting directly from implementation of a 
specific or generic practice. An integral part of verifying practice 
implementation. May be explicitly stated or implied by the 
practice statement or associated informative material." 


And from page II-110 


A "The use of informative material in the appraisal reference 
model to form a checklist is explicitly discouraged." 


And from page III-50 the glossary definition for direct artifact 


A fhe tangible outputs resulting directly from implementation of a 
specific or generic practice. An integral part of verifying practice 
implementation. May be explicitly stated or implied by the 
practice statement or associated informative material. " 
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Select the processes or subprocesses in the organizationG set of 
Standard processes that are to be included in the organizationG 
process-performance analyses. 
Select processes/subprocesses that will help us understand our 
ability to meet the objectives of the organization and projects, and the 
need to understand quality and process performance. These 
subprocesses will typically be the major contributors and/or their 
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n | d P pane Features 
Establish and maintain definitions of the measures that are to be 
included in the organizationG process-performance analyses. 
Select measures, analyses, and procedures that provide insight into 
the organizationG ability to meet its objectives and into the 
organization quality and process performance. Create/update clear 
unambiguous operational definitions for the selected measures. 
Revise and update the set of measures, analyses, and procedures as 
warranted. In usage, be sensitive to measurement error. The set of 
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SAUL A amel Lalo pm =o) ems) cele (ol-lee mm IL Com aColale-le| 


November 2 
Carnegie Mellon University 


measures may provide coverage of the entire lifecycle and be 


controllable. 


——— Software Engineering Institute | Carnegie Mellon 


Your complimentary 


use period has ended. 


Co mM 0 | ete Thank you for using 


PDF Complete. 
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ybjectives 


fear Features 


Establish and maintain quantitative objectives for quality and process 
performance for the organization. 


These objectives will be derived from the organizationS business 
objectives and will typically be specific to the organization, group, or 
function. These objectives will take into account what is realistically 


achievable based upon a quantitative understanding (knowledge of 
variation) of the organizationG historic quality and process 
performance. Typically they will be SMART and revised as needed. 
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Statistically managed) 


Process-Oriented 
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Establish and maintain the organization's process-performance 
baselines. 


Baselines will be established by analyzing the distribution of the data 
to establish the central tendency and dispersion that characterize the 
expected performance and variation for the selected 

process/subprocess. These baselines may be established for single 


processes, for a sequence of processes, etc. When baselines are 
created based on data from unstable processes, it should be clearly 
documented so the consumers of the data will have insight into the 
risk of using the baseline. Tailoring may affect comparability 
between baselines. 
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ablish Process-Performance 


pan Features 


Establish and maintain the process-performance models for the 
organizationG set of standard processes. 


Rather than just a point estimate, PPMs will address variation in the 
prediction. PPMs will model the interrelationships between 
subprocesses including controllable/uncontrollable factors. They 
enable predicting the effects on downstream processes based on 
current results. They enable modeling of a PDP to predict if the 


project can meet its objectives and evaluate various alternative PDP 
compositions. They can predict the effects of corrective actions and 
process changes. They can also be used to evaluate the effects of 
new processes and technologies/innovations in the OSSP. 
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Enrich the detailed process maps from CMMI ML3 to include 


executable process models developed from PPBs that summarize 
cycle times, processing times, available resources, sub-process costs, 


and quality. 
Identify key process handoffs during project execution in which exit and 
entrance criteria are important. 


At these handoffs, establish process performance models (PPMs) that 
predict interim outcomes. They govern process handoffs and provide 
leading indicators of problems with outcomes. 
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Next, identify controllable factors tied to upstream subprocesses that 
may be predictive of one or more of the outcomes (interim and final) to 


be predicted. 


Decide what type of data the outcome (Y) is and what type of data the 
factors (x@) are. 


Using the data types, begin to identify the statistical methods to help 
with our process modeling. (See next slide) 


(Other forms of process modeling are certainly possible.) 
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ANO A, Duniniy ariable Regression 


Type of Reviews Conducted; Type of Design Delivered Defect Density 
Method; Language Chosen; Types of Testing 


High-Medium-Low Domain Experience; Productivity 
Architecture Layer; Feature; Team; Lifecycle 
model; Primary communication method 


Estimation method employed; Estimator; Type of | Cost and Schedule 
Project; High-Medium-Low Staff Turnover; High- Variance 
Medium-Low Complexity; Customer; Product 


Team; Product; High-Medium-Low Maturity of Cycle Time or 
Platform; Maturity or Capability Level of Process; | Tjme-to-Market 
Decision-making level in organization; Release 


Iterations on Reqdés; Yes/No Prototype; Method of | Customer Satisfaction (as 
Reqds Elicitation; Yes/No Beta Test; Yes/No On- | aq percentile result) 
Time; High-Medium-Low Customer Relationship 
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Regression 


Reqds Volatility; Design and Code Complexity; | Delivered Defect Density 
Test Coverage; Escaped Defect Rates 


Staff Turnover %; Years of Domain Productivity 
Experience; Employee Morale Survey %; 
Volume of Interruptions or Task Switching 


Availability of Test Equipment %; Reqds Cost and Schedule 
Volatility; Complexity; Staff Turnover Rates Variance 


Individual task durations in hrs; Staff availability | Cycle Time or 


%; Percentage of specs undefined; Defect 
arrival rates during inspections or testing Time-to-Market 


Resolution time of customer inquiries; Customer Satisfaction (as a 
Resolution time of customer fixes; Percent of percentile result) 
features delivered on-time; Face time per week 
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Chi-Square, Logistic Regression 


Using these controllable factors... To predict this outcome 


Programming Language; High-Medium-Low Types of Defects 
Schedule compression; Reqdés method; Design 
method; Coding method; Peer Review method 


Predicted Types of Defects; High-Medium-Low Types of Testing Most 
Schedule compression; Types of Features Needed 
Implemented; Parts of Architecture Modified 


Architecture Layers or components to be Types of Skills Needed 
modified; Type of Product; Development 


Environment chosen; Types of Features 


Types of Customer engagements; Type of Results of Multiple Choice 
Customer; Product involved; Culture; Region Customer Surveys 


Product; Lifecycle Model Chosen; High-Medium- | Risk Categories of Highest 
Low Schedule compression; Previous High Risk Concern 
Categories 


= igh Maturity ow Do We Know 
==> Software Engineering Institute | CarnegieMellon ““Nevember2. 


2 Carnegie Mellon University 


Your complimentary 
use period has ended. 


= Cor ere Compete 
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Logistic Regression 


Inspection Preparation Rates; Inspection Review Types of Defects 
Rates; Test Case Coverage %; Staff Turnover 
Rates; Previous Escape Defect Rates 


Escape Defect Rates; Predicted Defect Density Types of Testing Most 
entering test; Available Test Staff Hours; Test Needed 
Equipment or Test Software Availability 


Defect Rates in the Field; Defect rates in previous | Types of Skills Needed 
release or product; Turnover Rates; Complexity of 


Issues Expected or Actual 

Time (in Hours) spent with Customers; Defect Results of Multiple Choice 
rates of products or releases; Response times Customer Surveys 

Defect densities during inspections and test; Time | Risk Categories of 

to execute tasks normalized to work product size Highest Concern 
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An example: 


A regression analysis was conducted to develop a statistically-based 
process performance model predicting Defect Density. 


As will be seen on the next slide, the regression model provides rich 


information about the role of the controllable x factors (Req’ts 
olatility and Experience) in predicting the _ outcome (Defect 
Density). 


In turn, this will provide management with rich information on how to be 
pro-active in changing predicted high levels of Defect Density to 
acceptable lower levels! 
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Regression Analysis: Defect Densi versus ReqtsVolatil, YearsDomainE 


The—egression @quation i3 
Defect Density = 0.484 + 0.480 RegqtsVolatility — 0.0242 YearsDomainExperience 


Predictor Coef SE Coef T PF | Prediction equation 
Constant 0.48367 fb. 03957 12.22 (7.000 for defect density 
ReqtsVolatility 0.47963 09511 5.04 /0.000 
YearsDomainExperience —-0.024215 6.001941 -12.48 \@.000 


p values below 0.05 
indicate the 
predictors to keep 
in the model 


5 = 0.00693207 R-Sq = 65.9% R-Sq({adj) = 64.6% 


Analysis of Variance 


Percentage of total 


Source DF 55 MS F Pp 


Regression 0.000 variation in defect 
Residual Errq Pp Vi density explained by 
Total indicates the model the model 


iS Significant 
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: | iP - Features 
Baselines and models, which characterize the expected process 
performance of the organization's set of standard processes, are 


22 


established and maintained. 


implementation. Our baselines and models incorporate our 
organizations and its projects, and allow us to create and execute 
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Central tendency and variation are the cornerstones of our 
understanding of these, allow us to understand risks in our 
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ablish the Projects Objectives 


oa Features 


Establish and maintain the project6 quality and process-performance 
objectives. 


These objectives will be based on the organizationG quality and 
process performance objectives and any additional customer and 
relevant stakeholder needs and objectives. These objectives will be 
realistic (based upon analysis of historical quality and process 
performance) and will cover interim, supplier, and end-state 
objectives. Conflicts between objectives (i.e., trade-offs between 


cost, quality, and time-to-market) will be resolved with relevant 
stakeholders. Typically they will be SMART, traceable to their 
source, and revised as needed. 


a igh Maturity ow Do We Know 
==> Software Engineering Institute | CarnegieMellon ““Nevember2. 


2 Carnegie Mellon University 


Your complimentary 
use period has ended. 


£ Co mM 0 | ete Thank you for using 


PDF Complete. 
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Expaliaa Features 


Select the subprocesses that compose the projectS defined process 
based on historical stability and capability data. 


The PDP is composed by: 
A selecting subprocesses 
A adjusting/trading-off the level and depth of intensity of 


application of the subprocess(es) and/or resources 
to best meet the quality and process performance objectives. This 
can be accomplished by modeling/simulating the candidate PDP(s) 
to predict if they will achieve the objectives, and the confidence level 
of (or risk of not) achieving the objective. 
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Analyze organizational goals and customer requirements to determine 
the desired outcomes in cost, schedule, and quality. 


Identify alternative process compositions. 


Evaluate these alternatives with respect to the desired outcomes in 
cost, schedule, and quality by: 


A Calibrating PPMs with distributional data from PPBs for key 
Subprocesses 


A Applying the calibrated PPMs to evaluate alternatives 
Select/Establish the appropriate process composition for the project. 


The next two slides illustrate this with an application of Monte Carlo 
simulation and optimization. The simulation and optimization help to 
confirm which choices we should make. 
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lect the Subprocesses that Will 
Managed 


Select the subprocesses of the project's defined process that will be 
Statistically managed. 


Subprocesses that are the major contributors to or predictors of the 
accomplishment of the project interim or end-state objectives will be 
selected. Additionally, these need to be suitable for statistical 
management. Statistically managing the selected subprocesses 
provides valuable insight into performance by helping the project 


identify when corrective action is needed to achieve its objectives. 
Select the attributes that will measured and controlled. 
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Monitor the project to determine whether the project objectives for 
quality and process performance will be satisfied, and identify 

! Compare the actual versus estimated and corresponding actual trend | 
i versus estimated trend. If wedée not meeting our objectives or based | 
! on the actual trend it looks like we wond achieve our objectives in the ) 
' future, document what we might do to fix the shortcoming/potential =! 
: shortcoming. 


Monitor the project 
A Manage stability and capability of selected subprocesses. 
A Track quality and process performance data including suppliers6 
A Update/calibrate PPMs and predictions based on results to date. 
A \dentify deficiencies/risks to achieving objectives (e.g., where 


current performance is outside tolerance intervals, or 
prediction/confidence intervals are not contained within 
specification limits). 
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Management reviews dashboards that include 
not only outcomes but leading indicators such 
as the controllable x factors identified in OPP 
and earlier in QPM. 
There are typically 3-5 leading indicators for 
each outcome (or lagging indicator) that may 
be used in a process performance model. 


igh Maturity ow Do We Know 
SAUL Amel Lalo pm =o) ome) exe (or-lee mm IL Com aColale-le| 


November 2 
Carnegie Mellon University 


2 


——— Software Engineering Institute | Carnegie Mellon 


Your complimentary 
use period has ended. 

Thank you for using 
PDF Complete. 


Seemed Features 


The blue lines 
represent the 
use of process 


models to criteria aaa 
statistical ly Reporting Periods 

predict — 
outcomes Success indicators 


(Lagging Indicators) 


Analysis 
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Management typically now only spends a fraction (20%) of each 
management review looking at the lagging indictors (e.g. the 
outcomes of cost, schedule and quality) 
Most of their time (80%) is instead spent reviewing the statistical 
management of controllable x factors and the predicted outcomes 
based on the x factors. 
Inherently, the discussion focuses on management pro-actively 
taking action based on performance models and control charts of 
igh Maturity ow Do We Know 
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The project is quantitatively managed using quality and process- 
performance objectives. 


* Explained in QPM goal 2 


Projects are managed through the use of: 

Aneasuring and controlling quality and process performance 
attributes. 

Astatistical techniques to ensure stable and capable subprocesses 
/PPMs to predict if objectives will be met based on current 


performance 
Aspec limits to indicate when the performance of current processes 


will adversely affect the project ability to meet its objectives 
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Select the measures and analytic techniques to be used in statistically 
managing the selected subprocesses. 


Identify the measures that will provide insight into the performance of 
the subprocesses selected for statistical management and the 
Statistical techniques that will be used for analysis. These measures 


can be for both controllable and uncontrollable factors. Operational 
definitions will be created/updated for these measures. Where 
appropriate (i.e., they are critical to meeting downstream objectives), 
spec limits will be established for the measures. 


igh Maturity ow Do We Know 
SUC Aamo l Lalo pm =o) ems) exe (or-lee mm IL Cm aColale-le| 


—— Software Engineering Institute | CarnegieMellon  ““Qycme2 


2 Carnegie Mellon University 


ply Statistical Methods to 


Your complimentary 


ang Dil use period has ended. 
EComplete nr comaite 
7 ; ane Features . . 

ariation 
Establish and maintain an understanding of the variation of the 
selected subprocesses using the selected measures and analytic 
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Selected measures for the subprocesses will be statistically 
controlled to identify, remove, and prevent reoccurrence of special 
causes of variation, or in other words, stabilize the process. When 
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Monitor the performance of the selected subprocesses to determine 
their capability to satisfy their quality and process-performance 
objectives, and identify corrective action as necessary. 
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! ' Compare the actual versus estimated and corresponding actual trend | 
1 versus estimated trend. If we@e not meeting our objectives or based ) 
! on the actual trend it looks like we wond achieve our objectives in the ! 
' future, document what we might do to fix the shortcoming/potential —; 
' shortcoming. : 


For a stable subprocess, determine if the control limits (natural 
bounds) are within the specification limits which indicates a capable 
subprocess. If it is not, document corrective actions that address the 
capability deficiencies. 
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Record statistical and quality management data in the organizations 


measurement repository. 


Record the data along with sufficient information to understand the 
context for the data and thus make the data usable by the 


organization and other projects. 
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process is statistically managed. 
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Selected subprocesses are statistically managed to ensure stability 

and capability (i.e., special causes of variation are identified, 
removed, and prevented from recurring and the control limits of the 
subprocess are kept within the specification limits). 
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That can be a valid business objective 


But, it is in all of our best interest to ensure that the number means 
something 


A That means paying attention to the informative 
A The richness of the model is in the informative 
A The ideas/concepts that add value are in the informative 


Without the informative material Levels 4 and 5 add little of even the 
minimum we all believe they are. 


If it is not value added, change it! 
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Misperception —1 


lf we measure more things, and involve more 
people in reviewing and using the measures, we 
will eventually achieve Maturity Level 4... 


The key to achieving high maturity is measuring 
the right things, and using the correct techniques 
to analyze and interpret the measures... 


We need to wait until we have more of the right 
kind of data before we can attempt to implement 
High Maturity Practices... 
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Misperception —2 


Adding the use of Control Charts to the practice 
of measurement and analysis results in Maturity 
Level 4... 


All | need to do is to use Control Charts to 
analyze the outcome of our critical subprocesses, 
and we can control them... 


Using threshold based on specification limits is an 
acceptable alternative practice for QPM... 
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Misperception —3 


My organization only has small short-cycle projects, 
SO we can never apply QPM... 


There is some minimum number of subprocesses 
(yet to be specified) that must be statistically 

managed before you can achieve 
Maturity Level 4... 
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Misperception —4 


All the things we need to 
understand about high 
maturity practices can 
be adequately explained 
in a single conference 
presentation or tutorial. 
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Instructive Examples: 
Recognizing Misinterpretation 


High Maturity Misconceptions 
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Tongue-In-Cheek Warning! 


The audience is reminded that the examples 
in the next four slides are intended to be 
instructive — not pejorative. 


MY ACCOMPLISHMENTS 
THIS MONTH INCLUDE 
COMPLYING WITH OUR 
I8S0 7000, SARBANES-— 
OXLEY AND SEI-S 
POLICIES. 


AND TF YOU MAKE A 
NELJ POLICY, T WILL 
COMPLY WITH IT SO 
FAST IT WILL MAKE 
YOUR HEAD SPIN! 


Is IT MY IMAGIN— 
ATION OR IS 
PRETENDING TO 
LJORK GETTING 
EASIER? 
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You Might Have Misunderstood OPP If... 


A table showing projected defects by phase looks like a 
Process Performance Model to you... 


The corporate average “Lines of Code Per Staff Day” by 
year looks like a Process Performance Baseline or a 
Process Performance Model! to you... 


A control chart used to ‘manage’ defects escaping into the 
field looks like a Process Performance Model to you... 


An Earned Value Management System seems to fulfill the 
requirements of Maturity Level 4... 
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You Might Have Misunderstood QPM If... 


“Tracking bugs across the lifecycle” looks like statistical 
management to you... 


You plan to “re-baseline” the contro! limits used to manage 
critical subprocesses on a quarterly basis... 


‘Management judgment is used to ‘adjust’ control limits 
used as thresholds to drive corrective actions... 


schedule variance and defect density look like perfectly 
good subprocesses to statistically manage... 
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You Might Have Misunderstood CAR If... 


You always respond to “Hi Severity” defects by saying 
“Let’s run a causal analysis and see what's going on”... 


Causal analysis is used only to find and resolve the root 
cause of defects... 


You dont see the value of applying DAR to select when 
and how to apply CAR... 


You don't see the value of applying CAR to select when, 
what and how to apply OID... 


You don’t see how Process Performance Models and 
Process Performance Baselines contribute to CAR... 
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You Might Have Misunderstood OID If... 


You think 42 Six Sigma projects — all focused on the 
inspection process — make a company Maturity Level 5... 


A 5% boost in the performance of a process that fluctuates 
by +/7% looks like a best practice to roll out immediately... 


The strength of an improvement proposal can only be 
measured by the persuasiveness of the author... 


You work-off improvement proposals only in the order in 
which they were received... 


You don’t see how Process Performance Models and 
Process Performance Baselines contribute to OID... 
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Standards for Interpretation: 
Expectations from Informative Material 
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Interpreting OPP —1 


Essential ingredients of Process Performance Models: 


¢ They relate the behavior or circumstance of a process or subprocess 
to an outcome (or a set of outcomes) 


¢ They predict future outcomes based on possible or actual changes 
to factors (e.g. support “what-if” analysis) 

¢ They use factors from one or more subprocesses to conduct the 
prediction 


¢ The factors used are preferably controllable so that projects may 
take action to influence outcomes 

¢ They are statistical or probabilistic in nature rather than deterministic 
(e.g. they account for variation in a similar way that QPM statistically 
accounts for variation; they model uncertainty in the factors and 
predict the uncertainty or range of values in the outcome) 


High Maturity Misconceptions 


—— Software Engineering Institute | CarnegieMellon __ will Hayes, November 2007 


© 2007 Carnegie Mellon University 


Interpreting OPP —2 


High Maturity organizations generally possess a 
collection of Process Performance Models that go 
beyond predicting cost and schedule variance, based 
on Earned Value measures. 


Specifically, the models predict quality and 
performance outcomes from factors related to one or 
more subprocesses involved in the development, 
maintenance, service or acquisition processes 
performed within the projects. 
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Interpreting OPP —3 


Requirements Architecture Design Code Test Release 


Delivered 
Defects 


Effectiveness 


2 ne Maintainable 
Data Brittleness 


Bayesian Belief Network Example from Bob Stoddard 


Fault Tolerant 
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Interpreting OPP —4 


Process Performance 
Models are often created 
dynamically in order to 
support ‘what-if analyses.’ 


Sampling and modeling 
methods are often used 
to populate Process 
Performance Baselines 
when historical data sets 
are small. 
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Not All Useful Models are PPMs 


Not all Statistical Models are 
Predictive Models 


Not all Predictive Models are 
Process Performance Models 


There are many valuable 
applications of statistics. 
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Predictive 
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Process 
Performance 
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Interpreting QPM —1 


The purpose is to enable proactive project 
management through use of: 


¢ Statistical management of critical subprocesses 

¢ Quantitative management of the project 
Retrospective analysis of aggregated data 
does not ensure proactive management. 


Understanding how current performance will 
impact downstream objectives is the point. We 
want leading indicators, not lagging indicators. 
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Interpreting QPM —2 
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Interpreting QPM —3 


If a subprocess is critical to the performance of 
a project, you probably want to measure more 
than one attribute of its performance. 


The term ‘variance’ is not intended to mean the 
difference between planned and actual. 


lf you dont measure variation, you can't apply 
Statistical management as intended in CMMI. 
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Interpreting CAR —1 


A quantitatively managed process Is intended 
when interpreting this process area. 


Specific Goal 1 implies a systematic approach 
to focus CAR activities, which relies on more 
than management or engineering judgment. 


There is a significant difference between using 
Causal analysis techniques at the lower levels, 
and satisfying the goals of this process area. 
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Interpreting CAR -—2 


Process Performance Baselines and Models 
are commonly used to focus on high-value 
problems and opportunities. 
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Interpreting OID —1 


A quantitatively managed process is intended 
when interpreting this process area. 


The difference between OPF and OID is more 
than the presence of additional data. 


specific Goal 1 implies a systematic approach 
to focus OID activities, which relies on more 
than management or engineering judgment. 
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Interpreting OID —2 


More than just gathering 
great ideas, the intent is 
to focus the search for 
innovation and maximize § 
the opportunity tohave §& 
impact. 
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"4 = There is great confusion about the meaning of the 
—, CMMI high maturity practices, even among lead 
j appraisers 
= Terms and concepts are confusing and ambiguous 


= Unclear how the practices and process areas relate to 
each other 


= Few examples presented, and no pictures 


" This presentation will offer a clear practical 
understanding of the practices, by presenting them in 
a more logical sequence, and by addressing the 
inherent interfaces between the practices 


NORTHROP GRUMMAN 
Rick Hefner, "A More Practical Set of High Maturity Practices" 
2 . | 
opyright 200 Northrop Grumman Corporation 


nt ti sint 


Project 


Organizational 


Causal Analysis and Resolution 
Organizational Innovation and Deployment 


Quantitative 
improvement 


5 Optimizing 


Quantitative 
management 


4 Quantitatively | Quantitative Project Management 
Managed Organizational Process Performance 


Requirements Development 
Technical Solution 

Product Integration 
Verification 

Validation 


Qualitative 
improvement 


3 Defined Organizational Process Focus 
Organizational Process Definition (for IPPD) 
Organizational Training 
Risk Management 
Integrated Project Management (for IPPD) 
Decision Analysis and Resolution 


Proactive 
management 


Reactive mgmt. 
(plan, track, and 
correct) 


Requirements Management 

2 Managed Project Planning 
Project Monitoring and Control 
Supplier Agreement Management 
Measurement and Analysis 
Process and Product Quality Assurance 
Configuration Management 


1 Performed 
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| want to use the organization’s 
ee standard process, but... 


. Does it’s performance and quality 
meet my customer’s expectations? 


. If not, how should | tailor the 
process? 
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Voice of the Process 
= the natural bounds of process performance 


Voice of the Customer 
= the goals established for the product/process performance 


Voice of the Business 
= process performance needed to be competitive 


Process capability may be determined for the 
= Organization 
= Product line 
= Project 
= Individual 


Typically, the higher the level of analysis, the greater the variation 
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at ol t r ani ation oto I t 
ro t ana r 


= Characterize the performance of the organization’s standard 
process Statistically 


= Develop models to help a project manager determine the 
performance they would be likely to get by using the standard 
organizational process model, given their project’s characteristics 


Historical Data Standard Process Baselines & Models 


Organization 
ss Project T 


customer and » NALS f2 
ei project : = 
—V a4 al objectives 
————— 


Project Results Tailored Process 


r ani ational ro_ ss 


SG 1 Establish Performance Baselines and Models 


Baselines and models that characterize the 
expected process performance of the 
organization's set of standard processes are 
established and maintained. 


elect Processes 
Select the processes or subprocesses in the 


measures that are to be included in the 
organization's process performance analy: 
SP 1.3 Establish Quality and Process-Performance 
Objectives 


process performance baselines. 
astablish Process Performance Models 
stablish and maintain the process __ 

performance models for the organization's set 
of standard processes. 
RICK Hefner, “A More Practical Set o 


FOF an 


Baselines characterize the “voice 
of the process”, based on the 
existing historical data 
¢ What is the current mean and 
variation? 


May need to subgroup the data 


Models allow projects to estimate 
their quantitative performance 
based on the historical data of 
other projects executing the 
process 


This helps them decide whether 
they need to tailor the 
organizational standard process to 
meet their project objectives, and if 
so, how 
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*Z = The organizational baselines show that the organizational 
— standard process is capable of a fielded product with a 

mean of 7.8 defects/ KSLOC with 30 control limits of 4.001 
to 11.60 defects/ KSLOC 


= The customer wants his 
software to have no more | Chart for Defects 
than 8 defects/KSLOC 


12 
UCL=11.60 
11 


= Does the project manager 
need to tailor the 
organizational standard 
process? 


= Does the Voice of the 
Process meet the 
Voice of the Customer? 


Mean=7.8 


LCL=4.001 


RF 
o fF aD N O O O 


0 5 10 15 
Observation Number 
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Establish Performance Baselines and Models 


Baselines and models that characterize the 
expected process performance of the 
organization's set of standard processes are 
established and maintained. 


Select Processes 


Select the processes or subprocesses in the 
organization's set of standard processes that 
are to be included in the organization's 
process performance analyses. 


SP 1.2 Establish Process Performance Measures 
Establish and maintain definitions of the 
measures that are to be included in the 
organization's process performance analysés. 

SP 1.3 Establish Quality and Process-Performance 
Objectives 
Establish and maintain quantitative objectives 
for quality and process performance for the 
organization. 

| SP 1.4 Establish Process Performance Baselines 

. Establish and maintain the organization's 
process performance baselines. 

SP 1.5 Establish Process Performance Models 
Establish and maintain the process 
performance models for the organization's set 


of standard processes. 
RICk Hefner, “A More Practical Set o 


Organizational objectives deal 
with improving overall 
performance or reducing 
variation (making the process 
more predictable), not setting 
“stretch” goals for projects 


The organization meets these 
goals by modifying the standard 
process, not driving the 
projects 


These improvements are 
handled via OPF or OID 
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Select Improvements 
Process and ECan eIe gy improvements that contribute 


to meeting quality an UBIaGes SpeUonmance 
objectives are selected. 


. SP 1.1 Collect and Analyze Improvement Proposals 
Collect and analyze process- and technology- 
improvement proposals. 

SP 1.2 Identify and Analyze Innovations 
Identify and analyze innovative improvements that 
could increase the organization’s quality and process 
performance. 

SP 1.3 Pilot Improvements 
Pilot pigcess and technology improvements to select 
which ones to implement. 

SP 1.4 Select Improvements for Deployment 


Select process- and technology-improvement _ 
proposals for deployment across the organization. 


lo nt 


These are proactive 
improvements driven by 
organizational objectives 
(based on business needs) 
beyond the current 
performance of the 
organizational standard process 


¢e Voice of Process does not 
meet the Voice of Business 
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*Z lunderstand the capabilities of the organization’s standard 
<7 process, but... 


_ 4... What are the project’s quality and process performance 
\ objectives? 
.. How should | tailor the process? 
.. What project subprocesses 
must be stable (predictable) 


and perform well, for me to 
meet my project’s objectives? 
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SG1 Quantitatively Manage the Project 


The project is quantitatively managed us} 
quality and process-performance objee 
SP 1.1 Establish the Project’s Objectives 


Establish and maintain the project’s quality 
and process performance objectives. 


SP 1.2 Compose the Defined Process 


Select the subprocesses that compose the 
project’s defined process based on historical 
stability and capability data. 

SP 1.3 Select the Subprocesses that Will Be 
Statistically Managed 
Select the subprocesses of the project's 


defined process that will be statistically 
managed. 


| SP 1.4 Manage Project Performance 


Monitor the project to determine whether t 
project’s objectives for quality and process 
performance will be satisfied, and identify 

corrective action as appropriate. 


nt oal 


Quality: defect levels of key 
work products or deliverables 


Process: productivity, efficiency, 
effectiveness of the project’s 
processes 


Rationale for how the project tailored 
the organization’s standard process, 
in order to meet their quality & 
process performance objectives 
¢ E.g., adding procedures to reduce 
variation 


Assumes the standard process 
includes subprocesses to select from 


Monitoring against the objectives 
established in SP 1.1 
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SP 1.1 Establish the Project’s Objectives 


Establish and maintain the project’s quality 
and process performance objectives. 


SP 1.3 Select the Subprocesses that Will 
Be Statistically Managed 


Select the subprocesses of the project's 
defined process that will be statistically 
managed 


SP 1.4 Manage Project Performance 


Monitor the project to determine whether 
the project’s objectives for quality and 
process performance will be satisfied, and 
identify corrective action as appropriate. 
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SP 1.3 Select the Subprocesses that Will 


Be Statistically Managed 

Select the subprocesses of the project's 
defined process that will be statistically 
managed 


=» Which processes do you need to be stable 
(predictable) in order to achieve your project’s 
objectives? 
= For these, eliminate special causes, characterize 
the process, and predicatively manage 


= The time needed to perform this practice is long 
and often unpredictable 


=» Many processes can not be made predictable 


= Example — objectives for delivered defects 
= Defect detection (peer review, unit testing, system 
testing) 
" Defect insertion (requirement definition, 
architecture, design, integration) 
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Tatlorin 


Organization 
Process Project Project Risk Supplier 
Tailoring Planning Tracking Management Management 
Quality Configuration Data Measurement & 
Assurance Management Management Analysis 


Project 


Process Project Project Risk Supplier 
Tailoring Planning Tracking Management Management 


li 


fly 
Assurance Management Management Analysis 


SP 1.2 Compose the Defined Process 
Select the subprocesses that compose the 


gow 
a 


2x 
ay: 


WW 


project’s defined process based on 
historical stability and capability data. 


a 


HHP 


NOR 
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SG 2 Statistically Manage Subprocess Performance 


The performance of selected subprocesses 
within the project's defined process is 
statistically managed. 


SP 2.1 Select Measures and Analytic Techniques 
Select the measures and analytic technic 


selected subprocesses. 


SP 2.2 Apply Statistical Methods to Understand 
Variation 


Establish and maintain an understanding 
the variation of the selected subprocesses 
using the selected measures and analytic 
techniques. 


SP 2.3 Monitor Performance of the Selected 
Subprocesses 


Monitor the performance of the selected 
subprocesses to determine their capability to 
satisfy their quality and process 
performance objectives, and identify 
corrective action as necessary. 


SP 2.4 Record Statistical Management Data 


Record statistical and quality management 
data in the organization’s measurement 
repository. 


nt oal 


Type of analysis to be performed 
(e.g., control charts) 


Key is understanding variation 
in the selected subprocesses 
(e.g., be able to compute 
standard deviation), NOT just 
metrics 


Given the stability and variation in 
the subprocesses, will we be able 
to meet our project-level quality 
and process performance 
objectives? 


This data is used to help select 
subprocesses in tailoring (SP 1.2) 
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SP 1.1 Establish the Project’s Objectives 
Establish and maintain the project’s quality 
and process performance objectives. 


SP 1.3 Select the Subprocesses that Will 
Be Statistically Managed 
Select the subprocesses of the project's 


defined process that will be statistically 
managed 


SP 1.4 Manage Project Performance 
Monitor the project to determine whether 
the project’s objectives for quality and 
process performance will be satisfied, and 
identify corrective action as appropriate. 


SP 2.2 Apply Statistical Methods to 
Understand Variation 

Establish and maintain an understanding of 
the variation of the selected subprocesses 
using the selected measures and analytic 
techniques. 


SP 2.3 Monitor Performance of the 
Selected Subprocesses 

Monitor the performance of the selected 
subprocesses to determine their capability 
to satisfy their quality and process 
performance objectives, and identify 
corrective action as necessary. 
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stions at [ 


What characteristics of the organizational standard 
process would be useful to understand? 


Which subprocesses would be useful to understand, 
for predictive purposes? 


Are these subprocesses predictable (stabilizable)? 


What data should the organization collect? 


To what level of detail should the organizational 
standard process go? 


What differences in project subprocesses are 
permissible? How do they impact the historical data? 
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*4 * It is possible to “go through the motions” of Level 4/5, 
— with realizing the business benefits 


i: tat] 
Sey 
Che) 


" An organization should recognize the purpose behind 
the practices and interconnections between practices 
= Characterizing performance of the standard 
organizational process (OPP) 
= Improving performance of the standard organizational 
process (OID) 
= Assisting tailoring decisions (OPP > QPM) 
= Characterizing performance of key project 
subprocesses for use in predicting project Success 


(QPM) 
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A OPP-SP 1.5 requires Process Performance Models (PPM) to 
estimate or predict the value of a process-performance measure 
from the values of other process, product, and service 
measurements 

A Chrissis, et.al., list four uses of PPMs: 


1. The organization uses them for estimating, analyzing, and 
predicting the process performance associated with the 


processes in the organization's set of standard processes. 


The organization uses them to assess the (potential) return on 
investment for process improvement activities. 

Projects use them for estimating, analyzing, and predicting the 
process performance for their defined processes. 

Projects use them for selecting processes or subprocesses for 
use. 
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A The technical approach is based on established methodologies for 
reliability prediction of software defect densities: 


Historical data is used to produce a process performance baseline 


The process performance baseline is characterized by key process 
attributes 


A Specific attributes are defined for each lifecycle stage 
AN talcum ol=yel|alaliale me) meatcm 6) c@)(-xeiem cal-maglele(c Mull 6) co10| (elm 0) ge lectors 


eleyaKe)aparelalexom ele tsio10 me lamlalome(svilaicrem-lie|elU](omvcel|Oloxs 


At the end of each lifecycle phase, predicted phase attributes are 
replaced by actual phase attributes, actual phase results are 
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New spec limit 
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Although Stable, the process was not capable of achieving 


new customer requirement NORTHROP GRUMMAN 
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Completed Causal Analysis and Resolution activity to determine 
common cause of excessive variation in upper control limit 


Developed an Action Proposal Plan for Improvement 
Implemented Action Plan 


Gauged Improvements by Tracking New Data Points Against 
Baseline 


Refined Analysis: Improvement vs. Correction AFTO 
Deployed Two 2005 AFTO Baselines (Improvement & Correction) 
Performed to New Baselines 


Updated Value Stream Map to Reflect TSSR Award Fee Criteria 
and Stable Baseline Performance 
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fo Features nts Instit t 
= TRAINING 
z = Provided awareness training of the new schedule requirements - 
a V, Series of email instructions and repeated face-to-face discussions 
y to train/brief personnel in TSSR award fee criteria vs. Production 


Contractual requirements 


= TOOLS 
= Improved AFTO tracking database to fRed-lightOAFTOs that 
exceeded planned days in critical phases of development 


= PROCESS 
= Inter-Directorate coordination to expedite revisions of wiring 
diagrams/schematic diagrams (WD/SD) impacted by AFTOs 


= Coordinated with Vehicle Engineering so updates of WD/SD associated 
with AFTOs would not adversely affect AFTO schedule 


NORTHROP GRUMMAN 


Ps SER MLB PROT 154 


Your complimentary 
use period has ended. 


# Complete —™nyoverssng 


few Features 


Days to Incorporate AFTOs 


Number of AFTOs by NG Rcv Date 


NORTHROP GRUMMAN 


Your complimentary | 
use period has ended. 


Complete "™22aai 


= Data confirmed that the process was 
performing within stable baseline limits 


| = Data indicated that process was not only 
below the UCL, but was under new spec limit 
(due to early improvement implementations) 


= Since much of the improvement plan had been 
incorporated by February 2 ,anewcontrol 
chart based exclusively on 2 data was run 
to analyze a tentative new baseline 
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» 4" Due to wide range of the upper and lower limits in the 
aA 2 process ( *t baseline), differences in distribution 
between Correction and Improvement AFTOs were not 
a factor 


"In2  ,with tightened performance limits, the 
differences between the two types of AFTOs became 
more evident 


= Further statistical analysis of the data would confirm 
that Improvement AFTOs and Correction AFTOs 
should be charted separately (i.e. there were two 
populations of data) 


NORTHROP GRUMMAN 


ah SER MLB PROT 154 


Your complimentary | 


use period has ended. 


ny C P OF et ray Thank you for using ro n t 


PDF Complete. 


Meese) features ; 


'4" Using Minitab, 2 new data points from 2 data were 
_. *; separated into two Subgroups 


) = Improvement Subgroup (38) 
= Correction Subgroup (24) 
= Each of the two Subgroups were then divided (binned) 


into a Contingency table 


= With data in a Contingency table, a Chi Square test 
could be conducted in Minitab 
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Chi-Square Test 


- £) Expected counts are printed below observed counts 
Chi-Square contributions are printed below expected counts 


: ‘ : ee Observed Counts 
1, 21 14 3 38 


26.35 9.81 1.84<———————— Computed Expected Counts 


1.088 1.793 iene ee 
2. 22 2 0 24 Chi-Square Contribution 


16.65 6.19 1.16 
1.723 2.839 1.161 Result Onlya . chance 
these differences occurred by 
natural variation. There is 
probability that there is a 
difference between 
Improvement and Correction 


distributions 


Total 43 16 3 = 62 


Chi-Sq = 9.338, DF = 2€-Value = 0.009 


two cells with expected counts Tess than five. 
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JCC = JIMIS Conformance Checklist 
AFTO = Air Force Technical Order 
JTF = Join Test Force 
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= St Goal Accomplishment 


4 = Technical Publications Established a Statistically Stable Baseline 
as for AFTO Incorporation Schedule 


= Demonstrated Performance to the Stable Baseline 


= 2"d Goal Accomplishment 
= Significantly Reduced the Process Performance Mean 
= Significantly Reduced Excessive Variation in UCL of the Process 
= Determined Need For Two Baselines: 
= Improvement AFTO Incorporation Schedule 
= Correction AFTO Incorporation Schedule 


= Customer Expectations of “Blue Rated” uality ave 
Been Maintained With Improved Schedule Efficiency 
= From Sept 05 Night Eyes 1 fDCMA is pleased to report that for 
JIMIS-JIMIS quality, Phoenix has maintained exceptional quality 
levels, currently 0.0 percent defect rate, through three consecutive 
revisions.O 
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_. #:= The improved process assures continued success in 
meeting and exceeding the customer’s defined 
schedule days for Routine AFTO Incorporation 


= The improved Disposition Incorporation of AFTOs 
process has a substantial impact on award fees 


= The total AFTO Disposition & Incorporation activity in 
TSSR represents 2% of the annual award fee 


= The improved and statistically stable AFTO process 
ensures that Northrop Grumman will continue to meet 
the requirements of the customer and realize of 
the Routine AFTO Award Fee activity 
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Sew Features 


= State of the industry 
= Our track record 


=" Key relationships between CMMI goals and practices 
and high maturity cost estimating behaviors 


= Practical advice on implementing high maturity 
behaviors 


=" Summary 


f" '= Who we are 
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The 
Process 
Company, LLC 


On sa an & On September 22, 2006, 
‘The Process Company, Completeeda | The Process Company, LLC Completed a 
SCAMPI Class A Appraisal k: a [ SCAMEI Class A Appraisal 
in Accordance with the Software Engineering Institute's Appraisal Fr ork in Accordance with the Software Engineering Institute's Appraisal Framework, 
using the SE/SW/IPPD/SS CMMI version 1.1 and determined . using the SE/SW/IPPD/SS CMMI® version 1.1 and determined that 
Northrop Grumman AGS and BMS : y F | oO rl d a Northrop Grumman Integrated Systems, AEW and EW Systems 

met the goals of 3 2 aes ———— met the goals of 

> ¢ SEI Level 5 Process aes ay SEI Level 5 Process Maturity 


Melbourne 
Z 
CMMI Level Agden Life LZ 


Andreas R. Felsi . 
SEI Authorized Lead Appraiser SEI CMMI Lead Appraiser 
The Le The Process Company, LLC 
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= Industry record is dismal 
= 2006 Chaos Report 
= 46% of projects are rchallengedowith cost or schedule overruns 
a, or requirements gaps 
y = 19% of projects fail 
| = Barry BoehmG data indicate a + 50% proposal accuracy is 
common 


‘a * Size (SLOC) 
+ Cost ($) 


0.5x 


Concept of a Product Detailed Accepted 
operations specification design design software 
0.25x specs. specs. 
LA A A A A A 
Feasibility Plans& Product Detailed Development and test 
requirements design design 


PHASES AND MILESTONES 


Sources: Chaos Report — Rubinstein, “Standish Group Report: There’s Less Development Chaos Today,” SD 

Times, March 2007. Boehm data — Pfleeger and Atlee, Software Engineering: Theory and Practice, 3 edition, 

Prentice-Hall, 2006; also published in Boehm et al., Software Cost Estimation with COCOMO II, Prentice-Hall, NORTHROP GRUMMAN 
3 2000. 
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4 = Root cause analysis is difficult to establish 


= End-of-job actuals are confounded with the project 
management track record 


= Credit, or blame, must be shared 


= Our track record 


= Seven major SW development projects completed 
1998-2007 


= Median SW cost performance index (CPI) = 102% 


= All projects completed on schedule with schedule 
performance index (SPI) = 100% 


= As-delivered SW quality at six-sigma levels 
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nriserati s 
« *)* “Must-win” estimating efforts 
; r, = Are planned and managed like projects 
f = Follow a defined process 

= Are executed by a team of product & estimating 

specialists 
" All relevant IPTs, engineering disciplines, and 
other stakeholders must commit to the estimate 

= Identify & involve (with mutual agreement) 

= Monitor & control 

= Objectively evaluate 

= Review status with higher management 
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_— Determining re ec " 
a Fe : Establish A What inwhaté out aes PM SG 2 Statistica 
g ' Estimates SW sizing spo Performance... 

| A SW estimate fer" es er 
Ensuring executability - ee 

A Schedule & staffing fe 
greens A Risk reducers <¢""F ‘OPP.SG_ Baselines 
A Reuse plan "Models 


* 
2 
Weegee Lo ke Seene 
"Seumnneeeee ee 


A SW build plan 


Herding the cats 


PP SG Obtain A System/Software/Test 


Commitment 


PP Project Planning 


PM uantitative Project Management 
OPP Organizational Process Performance NORTHROP GRUMMAN 
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A Estimating the process defines the 
process 


A The estimate depends on the process 
baseline, closing the estimating loop 


IPMSG_ Getting toa 
Defined Process 


A SW Cost Working Group 
A SW Process Management Team 


a, 
a %% 
« 


IPM SG 2 Working 
with Committed Team 
Mates 


%s 
Bay 
« 


ee 
geen er ee ag 
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se a, 


“SAM SG Supplier’ 
“., Agreements _.” 
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IPM Integrated Project Management 
SAM _ Supplier Agreement Management 
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Expanding the definition of 
fprocesso 
A Estimating with knowledge of 
process variance in SW size, 
cost, schedule, staffing, etc. 


A Confidence/risk predictions 
A Monte Carlo validation 
A Life Cycle Cost optimization 


PMSG _ The 
uantitatively Defined 
Process 


PM SG 2 Statistical 
Management 


A Providing a high maturity 
infrastructure 

A Enabling history & risk based 
estimating 


OPP SG Performance 
Baselines Models 


PM uantitative Project Management 
OPP Organizational Process Performance NORTHROP GRUMMAN 
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= No rogues 
= fWe don@ need no stinking process!0 
} = Manage the estimate 
= One is better than many 
‘+ ~— = WhoG on First? 
= Two is better than one 
= Parametric tools work 
= How do you use them credibly? 
=" Ensure executability 


= Think about execution risk. Your management and your 
Customer do 


= Risk items will be in the Customer evaluation of Most Probable 
Cost. Addressing them in the bid is up to you 


= Know how your Customer scores an estimate 
= Avoid the Lake Wobegon syndrome. You need cost realism and 
reasonableness 
= Justify, justify, justify reuse 


= Establish the pedigree and substantiate the choice 
NORTHROP GRUMMAN 
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ia Features 
ollo a In sti atin 
-_ «= SW Sizing procedure 
| jt = Allowable methods 
| = Counting rules 

7 = Reuse sizing 

= Checklists 

= SW Estimation procedure 


=" SW Cost Working Group 


= Parametric Model for size-based 
components 


= Discrete methods for other costs 
= Discipline review approval 
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SW Cost Working Group 


_aW Engineering A Responsible to produce a unified & integrated 
tr — > _ system/software cost model for the project 


A Rules of engagements for team members 
A Review of software technical and mgmt metrics 
j reported by each team member 
Nelatalce]e A Review of software estimates reported by each team 
Grumman member 
(SCWG A Review of cost modeling parameters reported by each 
Sy) team member 
A Coordination & communication among the SCWG 
members to ensure mgmt commitment by all 
A SW process issues 
A SW estimation risk 
A Issue and monitor estimating actions 


Team Mate | Team Mate | Team Mate | Team Mate | 


Our SCWG anticipates the SW Process 
Management Team that will oversee and manage 


the development after contract award. 
NORTHROP GRUMMAN 


ike} 


ISER-MLB-PR-07-143 


POF 


€ Comp ete 


System 
Requirements 
Analysis 
(2552) 


mee Features 


Your complimentary 
use period has ended. 

Thank you for using 
PDF Complete. 


a 


SEER SEM Estimated Tasks Indicated by Light ellow 


Background “(SEER)” 


System Design 


(.2. .2.2) 


S W Support 
(SEER) 


Govm’t Rqm’ts 


aetna ooo 


Requirements 
Management 
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Requirements 

Analysis 
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Type Ram 

Arisks 
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Products 
ACTRD 
ARAM/RTM 
AVCRM 
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AFunct Arch. 
AContext Diag. 


ASys Interfaces 


Design 
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ADefine 
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Synthesis 
Define & Refine 
AConcepts 


Aarchitecture 
composition 
Aallocations 
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Interfaces 
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Systems Studies 
Analysis ( .2. .2. ) 


ACAIV 


ASafety 


APerformance 


AT&s 
ACosite 
ASecurity 


S W Support 
(SEER) 
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sti 
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SEER SEM Categories 


SW Requirements 


SW Requirements 
Analysis (SEER) 
(eae a=. )) 


Develop CSCI 
Requirements 
e Requirements 


M S(SBA) 


JO .22..-.) 


Perform CAVE & 
CWIN Architecture 
& Design Modeling 
and Simulation 
AConstructive 
Avirtual 


ARequirement Management 


Asystem Modeling 


APerformance Monitoring 


AHMW & SM Design Oversight 
ATest Planning Support 


System Design Assurance ( .2.2. . ) 


ATech Manual Devel Support 
ATraining Devel. Support 
AData Analysis 

Asystem Level Doc. Updates 


Products Analysis 
APhyscl. Arch. e Initial Use Cases 
Asss e Activity diagrams 
AFRDs e SRS 

AIRSs eIRS 

Perform UML 

Modeling 

AScenarios 

AUse Cases 


SW Pre 


e Protocol state 


e Sequence state 


e Data Base Model 


Diagrams 
e Class (refined) 
e Internal 


e Deployment 
e Data Base 


: Review models 


Detail 
Design 


machine 


machine 


architecture 


Schema 


e Updated 


e Approved 
design 


e Source code 
e Reuse coding 
guidelines 
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Test ears 
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Estimate 
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Model Estimate 
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SEER SEM 


ApPlans 
AProc. 


Planning 
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SCIi 


Build) 


{ 
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Tested 
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rT anon 
In n nt sti ats 
'£ = SW Sizing 
= Good: multiple, independent reviews of all size estimates by 
the SCWG and third party ream of expertso 


= Better: independent estimates with the same technique (with 
reviews) 


= Best: independent estimates with different techniques (with 
reviews) 


“Team of 
Experts” 
Teammates Reviews 
Perform 
SLOC Estimate 
Lab 
Reuse 


Elifer-\ireya) 
Technical 


Requirements Perform 


SEER-SEM Software 
or Other Cost Estimate 
Model Runs Team Review 


Perform 
Take [=yeX-Vare (=a) Baselined SLOC 
aU layer iceya) Baselined Function Points 
Point Baselined SEER-SEM or Other Cost 


Analysis Model Parameters 


= SW Estimation | addressed on next slide NORTHROP GRUMMAN 
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—_|Tools or 


= ery accurate — when properly 
' « calibrated used 


y = Know your Customer preference 


Sat Soles 


Key PRICE S Input Parameters 


A Platform 
Management Complexity 

A Internal & External Integration 

A Utilization 

A Schedule Constraints 
Language & Productivity Factors 
Application Factors 

A COTS & Furnished CSCI Data 

Risk Parameters 


Government Funding Events System Description 
A TRD Requirements 
A Derived Requirements 
A Baseline Solution 


WBS, Software Build Plan, < 
and Software Reuse 


A CAIV Constraints 


System Description 


A ROM Estimate 
A SLOC Estimate 
Effort A Risk Adjustments 
alidated Sizing 


Size 


Single Team Parametric Model 
(Tuned to Demonstrated 
Performance) 


Key SEER-SEM Input Parameters 


A Experience/Capabilities 
A Development Support Environment 
Development Environment Complexity 
A Schedule & Staff Constraints 
Product Development Requirements 
A Reusability 
Target Environment 
A Confidence Level (50%-80%) 


Software —_ » 
Schedule = 


Validated Software BOE | 
Effort& — > 
Schedule WBS, Schedule, Effort 
by Labor Category 
v 


Demonstrated 
Performance 


Team Productivity 
Risk-Adjusted __Teammate_uotes | 
ESdinauoniigdel Design-to-Cost Constraints 


Recognize that your bid defines the nop GRUMMAN 
project’s process. 


16 ISER-MLB-PR-07-143 


Your complimentary 
use period has ended. 
Thank you for using 
PDF Complete. 


meg Features 


7 bids do not always make the most sense 
| —_ = If you are bidding mean performance, you are almost certainly not 
— es at 50/50 anyway 
= Management or Customer direction 
: 2 , or other bid strategies require process performance 


baselines that capture statistical variation in the process 


= Commercial parametric tools do offer these capabilities 
= Variable Risk/Confidence settings for parameters & estimates 
= Monte Carlo risk analyses 


Air Force policy is to estimate and fund programs to a 
Hours anid Tactical Simulation high (80-90%) confidence. That is to say, programs 
4007 are to be estimated and funded so that the total 
program costs for any given program would be less 
than the budget 80-90% of the time. Also, program 
milestones and program completion should meet the 
planned schedule 80-90% of the time. 


3207 


2407 


1607 


Sources: US Air Force Software Management Guidebook, V0.9, December 
2004. SEER-SEM screenshot on this page is from Galorath’s “Regional 
Tactical Simulation” example. 


807 
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Staff Levels 


_" Functional discipline reviews 61505 
_ g; © Estimation methodology _ 
= Process, metrics & 
performance baselines 307.53 1 
= Indirect & other non-project oe 
commitments == = j= || _ Batre 


Schedule Months 


= Non-Advocate Review (NAR) i 


= Ensure program is executable within cost and schedule proposed 
and do not expose the company to unacceptable risk 


= Is the program executable? 


= Independent Cost Evaluation (ICE) 


= Independent, objective evaluation of proposed costs, designed to 
assess the reasonableness of the bases of estimates (BOEs) cost 
risks associated with program execution, and the resultant financial 
impacts 

= Is the cost realistic? 
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©4 = Specific allowances in estimate 
a = SW growth 
= Holchin, Popp studies 
= Planned vs. unplanned growth 
= Build currency with incremental development 


= Maintenance of the SW baseline between completion of 
software integration & test and final system delivery to 
the Customer 


= Multi-site development 
= Multiple Site Development in SEER-SEM 
= Management Complexity (CPLXM) in Price 
= Security requirements 
SW Growth Data Sources: Holchin, “Code Growth Study”, March 1996 and Popp, “Calibrating Software 


Code Growth,” NAVAIR, February 2006, but see also the US Comptroller-General data in Stewart, Cost NORTHROP GRUMMAN 
Estimating, 24 edition, Wiley, 1991. 
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# = Customer funding profile 


= Compatibility of detailed SW Build Plan with 
availability of all hardware, software, and lab 
components 


= Traceability of the SW Build Plan to the IMS 


= CMMI maturity of all system software sites that are 
part of the development team 


= Managing the development team to have one unified 
system software development process 
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Fi 
lect ‘ CMMI Process Areas Prime Subs CMMI Process Areas Prime Subs 
a RW) 4 Level (continued) 
Requirements Management v v Organizational Process Definition — () 
Project Planning v v Organizational Training v () 
: Project Monitoring & Control v v Integrated Project Management for IPPD v () 
Supplier Agreement Management v v Risk Management v () 
Measurement & Analysis v v Integrated Teaming v () 
Product & Process Quality Assurance v v Integrated Supplier Management v () 
Configuration Management v ) Decision Analysis & Resolution v () 
Organizational Environment for ¥ () 
Integration 
Requirements Development v 
Technical Solution v Organizational Process Performance () 
Product Integration a Quantitative Project Management v ( ) 
Verification ’ (a ET 
Validation v Organizational Innovation & Deployment v () 
Organizational Process Focus v Causal Analysis & Resolution v () 
Notes 
(1) Subcontractor internal processes and IPT operations integrate with primeG@ processes. 
(2) System requirements are allocated by prime; subcontractors develop requirements at the configuration item (Cl) level. 
(3) All subcontractors integrate, verify and validate their products to the Cl or subsystem level; this includes integration of software Cls into 
hardware Cls or line replaceable units (LRUs). Prime integrates, verifies and validates at the system level 
(4) Subcontractors follow their own CMMI-compliant business processes. 
(5) Prime is responsible teamG process control and optimization. 
NORTHROP GRUMMAN 
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4" Statistical process control (SPC) reduces programmatic risk 


 @ = Gives superior insight into average performance and variability of 
| a the controlled processes 


= Higher confidence estimates 
= Enhances predictability and stability in executing the job 


= Enables proactive process improvement to meet management or 
Customer performance targets 


= Removal of feommon causeovariation from the process 


sial o | r i 
Electronic Meetings/Engineering Checks 


o ost o | 
Using Lognormal Probability Density Function 


J\ 


aay ars xX 
ro os ii ation an NT 
Electronic Meeting - Engineering Check 
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Traini an T T T T 
» | fr ee nin 8723 8769 8879 8872 
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vement Initiated - Stage 1 & 2 Training NORTHROP GRUMMAN 
C. 
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Estimate un-substantiated by supporting data. This 
definition includes un-supported engineering estimates and 
declarative statements (i.e., the xyz task will require three 
engineers for five months). 


Estimate not well correlated to, or substantiated by 
supporting data. In general, engineering estimates were 
based on the estimator experience and expertise is 
substantiated, the use of non-substantiated scaling factors, 
use of comparatives where relevance of comparative is not 
substantiated. 


Note: Past experience shows that engineering estimates 
receive no higher than yellow. 


Estimate supported by relevant comparable data from 
rsimilar programsoand/or validated parametric estimating 
systems. 


Estimate supported by relevant comparable data from 
“multiple similar programs.o 


Estimate supported by production experience and/or_cost 
trend data for “multiple programs.” 


Source: ESC Training material NORTHROP GRUMMAN 
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istory matters — Customers will not accept forecasting an 
unrealized productivity improvement 


Government Funding Events oe 
A TRD Requirements 
A Derived Requirements 


Baseline Solution 
WBS, Software Build Plan, A CAIV Constraints 
and Software Reuse 


Key PRICE S Input Parameters 


A Platform 
Management Complexity 

A Internal & External Integration 

A Utilization 

A Schedule Constraints 
Language & Productivity Factors 
Application Factors 

A COTS & Furnished CSCI Data 

A Risk Parameters 


System Description 

A ROM Estimate 

A SLOC Estimate 
Risk Adjustments 
alidated Sizing 


Single Team Parametric Model 
(Tuned to Demonstrated 
Performance) 


Key SEER-SEM Input Parameter: 
A Experience/Capabilities 
Development Support Environmefit 
Development Environment Compexi 
A Schedule & Staff Constraints 
Product Development Requiremekts 
A Reusability 
Target Environment 
A Confidence Level (50%-80%) 


Software ——— » 
Schedule 


Demonstrated 
Performance 


Validated 
Effort & 
Schedule 


Software BOE | 


I 
WBS, Schedule, Effort 
by Labor Category 
v 


Risk-Adjusted ___Teammate_uotes | 
Estimation Model Design-to-Cost Constraints 


“All the women are strong, 
all the men are good-looking, 
and all the children are above average" 
Garrison Keillor NORTHROP GRUMMAN 
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Source 


Functionality provided 
certifications 
In-house expertise 


Maturity 


Previous use 


Existing runtime 
environments 


Existing test procedures 


Portability 


Maintainability, reliability, 


quality 


support 


S 


Identify already known reuse 
candidates (Prime and 
Identify Subcontractors) 


candidate 
functionality 


for reuse Searches 


¢ In-house repository 
¢ SW reuse web page 


Reuse 
function 
software 

found?, 


1BY-SV-1 (0) 8) 
code 


Request 
approval from Choose best 
customer for reuse candidate 
iccTurstomer=tate fe et) 


Customer Process 
approved? Complete 


Complete SW 


 MCUSER OLS eS 


for potential reuse 
candidates 


Do any 
Candidates pas: 
initial reuse 
worksheet 

creening 2 


Complete criteria 
checklist for reuse 
candidates 


reuse candidates 
meet criteria? 


Run reuse 
candidates software 
through SEAL 


Reuse criteria checklist 
evaluation areas 


* Architectural compliance 


© Availability 

* Classification level 

* Complexi 

* Compliance to standards 

* Data rights 

* Degree of modification 

¢ Design (UML) 

¢ Documentation 

¢ Innovativeness 

¢ Interface design 

¢ DIl COE/DISR 

« Language written 

* M&S support 

¢ Maintainability 

* Openness 

* Organizational maturity 

¢ Requirements satisfied 

© Schedule risk 

* Security concerns 

¢ Supporting tools 

* Technology Readiness 
Level (TRL) 

¢ Testability 

© Timing and sizing 

¢ User satisfaction 


Don’t neglect COTS software & hardware. 
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laa , = SW reuse checklists 
= SW reuse worksheets 


\” = Software Evaluation Assurance Lab (SEAL) 
reports 


= Integration with other software in the project’s 
System Integration Lab 


uality, Security, Performance 
> 
ITY 
a wey ag Inspected 
ad Software 
bh 


Code Reuse 
Library/DB 


Graded in Each Category TRP-06-1263G 
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ACOTS, GOTS 
AShareware 
AFreeware —_—_—_> 


ASubcontractor SW 
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4 = CMMI goals and practices should be used to shape 
«= *) your engineering estimating process 


= Estimates should be planned and managed like 
projects 


= Parametric tools work 


= Tune them to your process performance models and 
baselines 


= Estimate must be executable 


= Use your process performance models and baselines to 
achieve the desired confidence level 


= SW reuse must be justified 


NORTHROP GRUMMAN 


(Ad 


ISER-MLB-PR-07-143 


Your complimentary 
use period has ended, 


Complete 27 


ea Features 


123 


Richard L. W. Welch, PhD 
Northrop Grumman Corporation 
(321) 951-5072 
Rick.Welch@ngc.com 
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Communications and information 
networks 


Intelligence, surveillance, and Space and ground satellite oy ey-Ve-1 ilo) al-w-lale M0] ¢) oLe) a me-¥-) aVd [e1-1-) 
Ie=Yexe) al ar-li-s-¥- La leqs) communications systems 


We innovate, integrate, and manage technology. 
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Convince other stakeholders process improvement 
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1 Convince other stakeholders management is convinced 
1 Convince other stakeholders process improvement helps them 


Convince Management 
| Share process improvement vision 
[i wa KO)YA(0 (eK =>.4 =I 0 aT 1 n@) 
A SEI data 
A Other companiesddata 
| Provide internal ROI 
A Harris examples 
A Other possibilities 
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Convince other stakeholders process improvement helps them 


; 
, 
, 
Convince Management 
Share process improvement vision 
Provide external ROI 
ASEI data 
A Other companiesddata 
make)Vle(om alicia arolmnce@)| 
AHarris examples 
A Other possibilities 
Convince Other Stakeholders 
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Convince Management of Payoff 


1 External Research 
ARead and Research 
A Attend SEI Courses 


AWS Tigcmerelarselicclaies 


| Internal Research 
A Develop a plan 
FN BY=N7-) (0) 0 310) 
(ie =o [0 lore lcs maar-lar-letclaal=vall 
| Start or improve your process group 
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The Quality of a System is Highly Influenced by 
the Quality of the Process Used to Acquire, 
DY-AV{=i[e) oer-larem\reliaieellam 


This Premise Implies a Focus on the Processes 
as well as on Products: 


[ea alismsur- We) ale prescir-le)[s)alovom e)a-lanlsyom [amanrclalelccceuelalare 


| Belief in this premise is visible worldwide in quality 
aale)v{svaalclalesmlamaarclaleirolerelalalem-valemcy-vav/ (cm aleLO Stators 
(exe a SOR) r-lalel-leels) 


| This premise is also applicable to development 


(Introduction to CMMI Version 1.2) 
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Processes are Not Well Defined and Improvised by 
Practitioners and their Management 


Process Descriptions are Not Rigorously Followed or 
Enforced 


Performance is Highly Dependent on Current Practitioners 
Understanding of the Current Status of a Project is Limited 


Immature Processes Result in Fighting Fires: 


{ There is no time to improve i instead Rage uoners are constantly 
reacting 


Firefighters get burned 
1 Embers might rekindle later 


(Introduction to CMMI Version 1.2) 
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Process Descriptions are Consistent with the Way Work 
sw ANeiUr- live Blears 


They are Defined, Documented and Continuously 
Tan} e)ge)v{cre 


Processes are Supported Visibly by Management and 
Others 


They are Well Controlled 1 Process Fidelity is Evaluated 
Vale tal ie)cexcre. 


There is Constructive Use of Product and Process 
WNWiKsyekselcslaatsval 
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Processes Enable you to Understand What is Going on 


People Develop their Potential More Fully and are More 
Effective Within the Organization 


By Defining, Measuring and Controlling the Process, 
Improvements are More Successful and Sustained 


Bi atcm Mm] <=)i|aelele miar-layaye) ¢)ce) e)ar-lc-mu m-1e1n)a(0)(010\Vammmrelalalle|elcxs 
and Tools are Introduced Successfully Increases 


Wite]ccm stelalclilmlalie)anarclilelamiswA\c-li(-le)iomlpmiarcwANblel0\s] myaelele 
SEI Technical Report, Performance Results of CMMI- 
Base Process Improvement (CMU/SEI-2006-TR-004) at: 


Alda OM /ANYATAYAUVASX =) elAnlUMo1e [0/4 @]0le)i(er-dke)atsycefelel 0 |aaloyalesyaCLom acy ele) at 
s/06trO04.html 


(Introduction to CMMI Version 1.2) 
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Process Improvement Start Up 
Reaching Next Level of Maturity 
Out of Phase Defect Removal 
Reach Higher Levels of Maturity 
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ROI Difficulty 


ROI Should be More Specific as 
you Move Up in Maturity 
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rLittle has been done to study the return on investment (ROI) 
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Stakeholders 1 of 2 


Convince Other Stakeholders Management Is 
Convinced 


royale) Vmaate lars les) palcval misma crs) ale mal 6) celerctsys 
improvement 


ro) ale) Vamenta lars (0is)aalsval mismolayie)aeljalemere)an)e)it-lalec 


Management should fund é and provide 


and review and patard process Betntellclatess 
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Relieves chaos 
Reduces defects and rework 
Facilitates improvements 
Saves time and money 


Toyota makes process improvement a way of life. 
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( Provide Internal ROI 

| Harris examples 

| Other possibilities 


ow Do We Get On The Road To Maturity poe orTy = NDIA Conference - 16 
assuredcommunications — SElPartner cpus Sg Nrensicansiapli iia 


S 


Your complimentary 
use period has ended. 
Thank you for using 


PDF Complete. HY 
fon 


mil tle mre Wee) ce)ale mm mcr-(0(c MU tame@nleluiar-lalem@rgcrelleliiiny 
Develop a Vision | What Will Motivate? 

| Fewer problem programs 

| Faster, cheaper, better 
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Determine Estimated ROI 
maicys\o10| Minicom mate lamcem\ir-lareleloipalcvall 
Be Determined! 
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Enhance the Current Processes to 
| Improve efficiency and value for all stakeholders 
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| Provide predictable program execution 
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Establish Process Group, if Possible 
Document Current Processes 
TaksiiiiUlilelareliv4om wa KO lectssslors 


Determine Process Improvement Goals 
| Short term goals (start with PP, PMC, MA) 


| Long term goals (add PAs, plan for SCAMPI) 
| Estimate ROI 


Prioritize Process Improvement Goals 


Develop Process Improvement Plan 
| Determine process improvement measurements 
| Begin measurement collection efforts 
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Quality 132% a 
Custenis 55% | -4% | 14% 

Satisfaction 
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CMMIL 2 2 Additional 

~~ Award Fee 
Achieved 
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Potential 
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Per Cent 


SWCMML2 SWCMML SW CMM L SW CMM L CMMIL 
Customer Satisfaction Continues to Improve 
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Improvements in: 
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Defect Density (Defects KLOC) 


Integrating PSPS™ and CMMI® Level 5. Gabriel Hoffman, Northrop Grumman IT . May 1, 2003 
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Results 
met 25+ milestones In a row Improvements in: 


earned a rating of reExceptionaloin every 
applicable category on a formal Contractor Schedule 
Performance Evaluation Survey cycle time 


Customer 
Hours Invested: 124 in Defect Prevention (CAR) Beer 


(OLU]ESHEST- NV 10 Ma Kol) 0 a(0]U] fo (RoW (0) 0] 5m o\-1eme(=1(-101)) 
ROI 13:1 


Integrating PSPS™ and CMMI® Level 5. Gabriel Hoffman, Northrop Grumman IT . May 1, 2003 
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(, PROCESS IMPROVEMENT 


Support Expenditures $207,000 
$69,000 
=>.(=1010 1 (6) AO) im | ats) Ol =101 (0) A\S MINS Soret om 0010) 


$634,000 
Savings from Inspections | $2,524,000 
Total Cost Savings xe tsio] 00010 
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Process Maturity Profile 
by Reporting Organization Categories 


gw Commercial/In-house: 100% = 496 
@ Contractor for Military/Government: 100 % = 248 


@ Military/Government Agency: 100 % = 38 
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Not Given Initial Managed Defined Quantitatively Optimizing 
Managed 
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Program Analyzed Metrics 

Program Made Process Improvement Changes 
Process Changes Improved Metrics 
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Original zone of 
quality control 


Chronic waste 
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Statistics 
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results 


processes 
Address “Special Causes” 
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Continual measurable 
process improvement to 
meet business objectives 
Incremental innovative 
process improvements 
Address “Common 


Causes” 
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Objective: rMonitor key functional processes for predictability and 
measurable improvement (e.g., productivity, [Ure llinva le) 
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* not enough data points to classify as statistical control but can use trial limits 
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After process provements Taal e)(etaateyaiccze. 
Average cost performance improved 10% 
Lower end of predictable performance range improved 33% 
BENEFITS: Higher productivity, cost stability, higher award fees 
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Rework ROI 

| Estimate rework cost 

| Estimate defect reduction 
1 Estimate cost avoidance 


UPS ROI 

| Estimate cost for downtime 
| Determine cost of UPS 

| Estimate savings 
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| Estimate manual effort 

| Estimate automated effort 
| Estimate savings 


Other Suggestions? 
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Harris Process Compliance Monitor 

| Monitors all required processes 
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To be appraised at a later date (i.e., the process has not yet been 
executed by the program and cannot be appraised) 


Not Outside the scope of the project (e.g., Code and Unit Test 
Applicable Process is not applicable to a production-type program) 


Not Scored Pending an appraisal 


Fully Direct artifacts are present and appropriate (Note 2) 
Implemented No weaknesses noted (Note ) 


ASSESSMENT 
STATUS COLORS 


LI Largely Direct artifacts are present and appropriate (Note 2) 
Implemented One or more weaknesses noted (Note _ ) 


Partiall Direct artifacts are missing in the initial scoring audit or direct 
ee artifacts are present but inadequate (Note ) 
Implemented 
One or more weaknesses noted (Note _ ) 


Not Direct artifacts are missing for more than __ days from the initial 
Implemented scoring audit 


Note 1: A weakness ("gap") is considered if it is an impact to or risk of implementation of the process statement 
Note 2: An appropnate artifact is the IPM Expected Artifact or equivalent that demonstrates implementation of the process statement 
Note 3: An inadequate artifact does not demonstrate implementation of the process statement 
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ARepresents overall process Project Evidence 
compliance score for program This page allows users to edit project evidence. 
ABased on lowest color score i harsh, 
but in keeping with CMMI standards Baseline: Rev.45 27-Jul-05 


{ B LI Project Score Process: 


; Een ee Phase: 
ADepicts scoring distribution over all Scoring Distribution 


process items Expected Artifad 


NiO Pl OLE 20 Fl: 56 NS: 485 NY: 19 NA: 


AMore insight on overall project score eS 36 Total: 616 Rescore: 60 Project Artifact: 


Artifact Location 


Comment: 


ADepicts score for each process 
executed or being executed by this 
program 

A3 columns identify types of processes 

Alin PCM, point+click on underlined ISK 
acronym drills down to scoring details " Select Appraiss 
for process Val ® Display Alll § 


Export Project Status 
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‘Community 


Practical Software and Systems Measurement 
SEI Training Courses on Process Improvement 
Six Sigma Training 

Provide Technical Expertise 

Train Them to: 

| Follow the processes 

(in @Xo)|(=Yo1 mere )alsisicslalmanieyotsielesss 
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| Suggest improvements 
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Process Improvement Proposal: 
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Current Estimates for Creating Metrics and 
Reports Monthly ~$1.5 million per Year 

Current Estimate for Incorporating BI ~$2M 

Time Saved Each Month Can be Used Elsewhere 
©) tate} olelaroviies 

| Consistent, timely, accurate data 

| Flexibility in reporting 

| Easier new analysis 
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| State, Share and Spread the Vision Of Process 
Tan]e)ae)-laatesal! 


Wikelarslo[slealclalaya\el kom =i-bs\c1e mela mialc ms {eliie)pammlalc 
| Show and Tell how process improvement can 


improve their business (ROI) 


Other Stakeholders Tend to Follow Management 
fim Oxo) an falex-M O14 a\-) mem) €-1,¢-) ale) (eC) e-mearclmantclarclelsiparcyalt 

swore] ahvialexcre r-la lem smuirelcevaliale) 
Teams Want Improvement 


| Show and Tell how process improvement will help 
them (ROI) 
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Debra J. Perry 
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Division Process Group 
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debra.perry _harris.com 
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arris Corporation http www.harris.com 
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Melbourne, Florida 2 2- 
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Page 9-12 Introduction to CMMI Version 1.2 


http://www.sei.cmu.edu/products/courses/p44b.h 
idaalt 
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http://www.stsc.hill.af.mil/crosstalk/200 7/01/0701 
Reifer.html 


Page 15 No Satisfaction at Toyota 
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Page 20 - Performance Results of CMMI®- 
Based Process Improvement 2006 
http://www.sei.cmu.edu/pub/documents/06.repo 
rts/pdt/O6trO04. pdf 


Page 21 - Lockheed Martin results 


http://www.sel.cmu.edu/activities/cemmi/results/p 
dfs/2003-CMMI-016.pdf 


Page 22 - Integrating PSPS™ and CMMI® Level 
5. Gabriel Hoffman, Northrop Grumman IT . 
May 1, 2003 
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A Page 24 - Case Study of an Improvement Program 
Featuring Reviews and Inspections, by: Robert 
MacFarland, Ericsson Ltd.; American Society for Quality, 
Software Quality Professional, June 2001, Vol. 3, No. 3 


Page 257 Process Maturity Profile CMMI v1.1 SCAMPI 
v1.1 Class A Appraisal Results 2005 Mid-Year Update, 
oY] 0](-10 01 6\- 1674010 lo mm alii OM /AVIUAWAVVASL 2) OlANLUM=10L6/c-10)e)folistoUes 
program/profile/pdf/CMMI/2005sepCMMI.paft 


/, Page 26 - SEI CMMI Published Appraisal Results 
http://sas.sel.cmu.edu/pars/ 
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Page 34-35 rGetting There: Tips and 
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Maturity Level 4 or 59 by Gary Natwick, Debbie 
Perry and Sophie Boyd, SEPG Conference 26- 
29 March 2007 
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Capability 


Process Not Understood | Stable and Optimized 
Capability Quantitatively Controlled 

Process Proactive Predictive (Xe) al iiaelelels 
AN e) 6) fey= (era) Improvement 


Improvement | Project D)\Wiksile)aur- late D)\Wiksile)aur- late 
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Measurement | Monitor and Decision Systemic 

meleeis (Ore) alice) WiFelalale) KSSSUTESS 
Statistical 
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Practices ASSOLE Improvement 
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Benefit 
Category 


Cost Avoidance 


Productivity 
Gains 


Faster Time-to- 
Market 


Quality 
Improvement 


Estimated ROI 


Minimum Time 
(to achieve ROI) 


Other benefits: 

= Improved 
customer 
satisfaction 
Improved 
competitive 
positioning 

* Other 


yement Example 1° 2 


Benefit Range/Time ($ saved/months to realize)+ 


2 to 12% 
savings/ 
18 to 20 
months” 


5 to 10% 
annually * 


Not applicable 
during startup 


Not enough 
data 


15 to 51%! 
18 to 20 
months 

18 months 


Fewer 
customer 
complaints 
Perceived 
competitive 
gaps closed 


(edge) il(=ssie) im =A.o] om @4\VINV 1 @)eer-lalyzelile)als) 
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Reaching the 


8 to 18% fewer 


Optimization 
Next Level and 
Maintenance 
3 to 16% 
savings! 
16 to 18 
months 


8 to 18% 
annually 
Improved 
ability to 
predict/meet 
schedule 


Improved 
ability to 
predict/meet 
schedule 
12 to 26% 
fewer 
errors/post 
release 
12 to 36%/ 
annually 


errors/post 
release 


18 to 103% 
15 to 18 
months 

15 months Performed on 
an annual 


basis 


Continued 
customer 
praise 
Continued 
commitment to 
process 
maintained 


Increased 
customer 


Perceived 


competitive 
gaps closed 


assuredcommunications™ 


Out-of-Phase 
Defect Focus 


Finding 
escapes 
results in 6 to 
8% savings! 
annuall 
1 to 3% 
annually 
Improved 
ability to 
predict/meet 
schedule 
18 to 30% 
fewer 
escapes 


24 to 138%/ 
annually 


Performed on 
an annual 
basis 


Customer 
views you as 
best in class 

Perceived 
competitive 

advantage 


SEIPartner 


IS 


NDIA Conference - 46 
12-15 November 2007 


Your complimentary 
use period has ended. 


+E =P DE te a you for using 
p DF Complete. yement Example ae) 4 HARRIS 


Click Here to upgradete 
Unlimited Pages and EXiag=ueeenanaes 


Scenario Range of Cost/Time 
$ expended/months to complete)+ 
Large 


$1.5 to 2.5M! $2.5 to 3M/ 


‘Starting Up $1 to 1.5M/ 


18 to 20 months | 18to 22 months | 20 to 24 cons 
Reaching the Next Level €0.75to iM | $4 to 1.5M/ $1.5 to2 
in Process Maturit 1i2to 16 months | 15 to 18months | 16 to 21 Faun’ 
Optimization and $0.35 to 0.5M/ $0.5 to 0.75M/ $0.75 to 1M 


Maintenance 12 months++ 12 months++ 12 months++ 


Out-of-Phase Defect $0.5to0.78M/ | $0.78 to 1.0/ $1.0 to $1.3M! 
Focus 12 months++ | 12 months++ 12 months++ 


——— 
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Network Centric Systems, McKinney, TX | 


CMMI Level 5: Return on 
Investment for Raytheon N TX 


Donna Freed 


KARRI 


Envision Improvement 


Operational Results 


Integrated Product 
j Development System: 
-"| Where we define our product 
development processes 


| eae Product Teams: 
ross-functional resources to fv. 
implement our processes 


Raytheon Six Sigma: 
How we improve our 
processes 


“yf | Programs Integrate R6o, IPDS ‘ 
and CMMI into their Pland 
| 


Ric 
p 4 - 


Capability Maturity 
Model Integration: 
The yardstick for 
judging the maturity of 
our processes 


Product teams | use common tools and | processes in an environment. of i 
continuous improvement guided b ir lustry “Best Practices” 


Raytheon 
Improvement Results 


¢ Achieving CMMI Level 5 Certification for Raytheon image 
and competitive advantage is one thing, but look at the 
operational results. 


“Meeting reece all pened concurrent with SEl 


—CPI by 5 percentage points, and reduced variation by 34%. 


—SPI by 8 percentage points, and reduced variation by 50% 


—Defect Density by 44 percentage points, and reduced 
ariation by 31% 
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* Demonstrated the linkage between R6o and CMMI Levels 4 & 5. 


* Characterization included over 300 applications of R60 tools such 
as ANOVA, cause and effect, regression analysis, histograms, Cpk, 
hypothesis testing, logical process mapping, and others. 


* Identified five projects to reduce variation in organizational 
performance and support the CMMI Level 5 timeline. 


¢ Enabled CMMI Level 5 certification. 
— Improvement of Business Performance was recognized by Assessment 
Team as global strength in the CMMI Level 5 Assessment. 


* Contributed ROI of 3:1 through significant cost avoidance realized 
organization improvements 
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To Be Top-Tier is to See With New Eyes 


= Process improvement is built into the system 


» Evidence Books used as patterns from previous 
appraisals were not sufficient to meet later 
expectations 


= Needed to add more evidence as our understanding 
of what makes a good process has grown 


= The culture has changed 


» Process improvement is the object of many CAR and 
Six Sigma projects 


= Process people are not the first to go when budgets 
are cut 


= It gets easier each time 
» Familiarity leads to quicker startup 
= Less training needed, less resistance to change 


Project Leaders Gain 
ie More up front thinking means less work later al 


= Fewer problems and risks along the way 


= Improved processes added slack to cost and 
hedule curves 


= Fewer repta 
« Easier to give back resources 
= Easier to help other projects 


= Other projects consulted us to find out why things 
were going so well 
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Projects Gain 


= Produced more value-added products with reduced 
effort and time 


= Instead of overrunning budgets and schedules, 
products are delivered early and on budget 


: Meshed well with cost reduction efforts 


Easier to understand the role of Systems 
Engineering in Software Development 


» Better understanding of how to get job done 
= Less stress 
» Less time doing rework 

Easier to transfer from project to project 


Easier to understand need of Systems Engineering 
in Software Development 


Concerns were escalated more quickly to the proper 
level of attention 


More enthusiastic about looking for improvement 
opportunities 


= Down side: SPIN meetings are much less popular 
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Industry Median 
of Organization 


Northrop-Grumman (Metbourne) 


of Engineering 


Where does you company fit? 
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Understanding CMMI 
Measurement Capabilities 
& Impact on Performance: 
Results from the 2007 SEI 
State of the Measurement 
Practice Survey 
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Basiect Features 


Purpose & scope of the survey 
A The respondents & their organizations 


2 


Results 
A Measurement resources & infrastructure 


A Value added by measurement 


Understanding CMMI Measurement 


Capabilities 
Dennis R. Goldenson, 14 November 2007 
© 2007 Carnegie Mellon University 


A Software measures used 


A Data quality & integrity 
A Organizational perspectives on software measurement 


Summary, lessons learned & next steps 


——— Software Engineering Institute | Carnegie Mellon 
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fe Be CC! Features 

Careful & well executed use of measurement & analysis 
A |Is awell accepted tenet in many fields of endeavor 


3 


A Including of course CMMI 


Basic aims 
A To inform management & technical decisions based on empirical evidence 


Capabilities 
Dennis R. Goldenson, 14 November 2007 
© 2007 Carnegie Mellon University 


A &to judge the results of those decisions once made 
But, how well, and how frequently, are measurement practices put into 
Understanding CMMI Measurement 


effect in our own field? 
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Benchmarking: The current state 
Some professional & consulting organizations maintain repositories they 
use for establishing benchmarks & facilitating benchmarking activities 
However, their measures & measurement definitions differ in many ways 
In that sense, one cannot speak confidently about hAndustry standardso 
Consortium {as described at last year&é CMMI Technology Conference} 


A 
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A 


A 
Which ts why the SEI has launched the Performance Benchmarking 


6 Updates of trends in typical use of measurement in software & systems 


A Aim to provide data that's not yet widely available 
Understanding CMMI Measurement 


A 
The state of the practice surveys 
engineering 

6 To help projects & organizations judge their progress relative to others 

A But there also will be a continuing need to track qualitative as well as 
Capabilities 

Dennis R. Goldenson, 14 November 2007 


quantitative descriptions about the quality & frequency use of measurement 


in our field 


——— Software Engineering Institute | Carnegie Mellon 
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Basiect Features 
A Screening question to identify respondents whose organizations develop 


23 SEO ate 


New this year 


software but rarely if ever do measurement 


A Questions about 


6 Resources & infrastructure devoted to measurement 
6 The kinds of measures used by the responding organizations 


6 Practices to ensure data quality & integrity 


Capabilities 
Dennis R. Goldenson, 14 November 2007 
© 2007 Carnegie Mellon University 


6 Value added by doing measurement 
Among other things, these questions allow us to make some useful 
Understanding CMMI Measurement 


comparisons by CMMI maturity level 
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15' survey described at last year's CMMI technology Conference 


zesioC! Features 
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Similar results this year 
A Moderately strong relationships exist when comparing the replies of 
respondents based on: 
Management versus staff roles 
Industry versus government organizations 


6) 
6) 


Capabilities 
Dennis R. Goldenson, 14 November 2007 
© 2007 Carnegie Mellon University 


The United States versus other countries 
Understanding CMMI Measurement 


6) 


6 Organization size 
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Test SPCRs, Issues 
¢ Peer review defects 
¢ SPCR s (software (sw) /sys problem 
change requests) 


¢ Developmental 
¢ Formal 


¢ Severity 
— 1 to 5 (most to least severe) 
¢ Issues 


11/28/2007 


Ww) 
i 
O 
OW 
ep) 
ow 
oO 
2 
co) 
QO 
we 
oO 
D 
OQ 
= 
= 
= 


Finding Defects* and SPCR s* 
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Number of Peer Reviews by Discipline by Year 


GeEng 

w Controls 

O Systems 
Oo Software 


2001 2002 2003 2004 2005 2006 2007 


> 
@ 
i 
_ 
@ 
@ 
i BS 
ro 
Oo 
_ 
@ 
ae 
& 
= 
= 


Peer Review Process Measure 


¢ ‘Goodness: 
— Number of defects found/missed 
— Early identification of defects 
— Predictability of number of defects 
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Number of Peer Defects 


Software (Sw) Defect Origin 


Origin of Defects found in Peer Reviews 


2002 
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2003 


2004 


2005 


2006 


2007 


System Requirements Analysis 
B System Architectural Design 

O Software Requirements Analysis 
O Software Preliminary Design 


@ Software Detailed Design 
G Code & Unit Test 

B Software Integration 

O Software Qualification Test 


@ System Integration 
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2002 Sw Peer Review Defect Detection vs. Origin Phase 


Sw Software Qualification Test 
Qualification 


Sw 
Software iptegration 
Integration 


GS Software Integration 
Ge Pr ary Desig 


Code & Unit Test 


: Code & 
£ Unit Test 


Software Detailed Design 


Sw Detailed 
Design 


Software Preliminary Design 


Sw Preliminary 
Design 


Software Requirements Analysis 


Sw Requirements 


ESCAPES 


Analysis | SSE SPER RTP IIB DET 
Number of Defects 


Detection 


Phase 
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Sw DEFECT ESCAPES FROM 
2003 PEER REVIEWS 2004 


Oetect Origin by Phase Ame 


Defect Origin by Phase 2003 
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Sw DEFECT ESCAPES FROM PEER REVIEWS 


Defect Ovigin by Phase 2006 


2006. 


Code & Unit Test 


Defect Origin by Phase 2007 


Code & Unit Test 
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2002 Sw Defect Escapes from Code & Unit Test 


Software SPCR Detection Phase & Origin Phase 2002 


Software Qualification Test 


Phase 


3 Software Integration 
WwW 
itegration 


Number of SPCR s 
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Sw Defect Escapes 
from Code & Unit 
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Number of SPCR 5 
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Software Issues by Year 
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Estimated Total Errors: | Selected Peak Location: & 
Projected Latent Errors: | } Error Discovery Efficiency: 39.35% 2003-2004-2005 
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SWEEP v4.1 - Project2 - Error Discovery Data and Rayleigh Fitted Histograms 


Software Preliminary Design 
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Process Benefits 


¢ Better management of defect detection 

— Defect discovery information 

— Defect and SPCR escape information 

— Identification of integration problems 

Fewer issues and shorter time to closure 
for issues 

Defect and SPCR predictability 

More disciplines performing peer reviews 
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Process Improvement Benefits 


¢ Peer Reviews (supports goal performance and quality goals) 
— Training to promote consistency in execution and recording of data 


— Results of root cause analysis pertaining to types of reviewers, 
number of reviewers, amount and nature of peer review material, 
duration of peer review, verification of data 


¢ Testing (Suppors Quality goal) 


— Made more formal; systematic analysis of software structure to 
determine most effective test cases 


— Analysis of peer review escapes to test 
¢ Build Structure (Supports Productivity goals) 


— Releases organized as sequence of builds, each of which addresses 
specific life cycle phase product requirements, and each of which 
has a full life cycle 
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Backup 


Severity Occurrence 


Software Peer Review Defect Severity by Year 
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‘ROI Objectives 


Use ‘ROI’ as a measure for 


— Understanding and quantifying the benefits/losses 
of Engineering Process Improvement (EPI) and 
Engineering Process (EP) performance for the 
Program 

¢ Support decisions 
— planning EPI 
— Performing EP 
¢ Increase efficiency of EPI and EP 
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I-MR Chart of Number of Defects per Software Det Dsg & Code PRs 


Individual Value 


21 26 
Observation 


Moving Range 
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Team Peer Review Screen 


= New Peer Review Master Record - Microsoft Internet Explorer 


New Peer Review Master Record Team 2 Pro f 


New | Save [| Cancel | G Options | 
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Software Size 


¢ Our example dealt with defect escapes overall 

¢ If we compare attributes of peer reviews of various 
work products of varying size, we normalize by size. 

¢ We also typically run several types of control charts. If 
Defects per Unit of size or time are used we include 

U charts 
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Today we'll discuss... 


= The problem of protecting 
due-date performance 


= How to “calibrate” your 
Project Planning Process, 
that is, determine the 
systematic error 


= Mechanically, how to get 
data from Microsoft® 
Project to Minitab® 
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Often projects have difficulty 
finishing on time 


Often projects have difficulty 
staying within budget 


Often agreed to scope or 
specifications are cut from a 
project to maintain cost or 
schedule 


All result in Customer 
irritation and perhaps 
undesirable consequences 
for the Project Manager 
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aiid Complete Projects 


on Time (to be 
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Add protection to 
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Project Manager 
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for Short Lead 
Time 
(quick response) 


Don’t add 
protection to tasks 
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i, defined requirements 
” = Competing priorities a 
= Clients changing their a 
mind 


= Unforeseen events 
(Murphy) 
= Poor communications 


= Unsatisfactory means of " 
measuring progress 


= Key people not available 
when needed 


Pressure to meet unrealistic 
due dates 


Factors outside our control 


Essential information not 
available on-time (designs, 
specifications, materials 
and authorizations) 


Too much re-work 


Lack of leadership or good 
management 


Need “protection” from these factors NORTHROP Gruseancany 
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This is not intended to be a 
dissertation on Critical Chain, just 
borrowing some ideas 


= Eliyahu Goldratt in his Critical 


Chain theory suggests that 
projects create buffers to protect 
due-date performance 


Project buffers are “zero- 
resource-tasks’” in schedules 
that absorb the risks inherent in 
planning 


The issue for the 

Project Planning Process is to 
determine how big, and where 
to place these buffers in a 
project schedule 
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minimize slack or float, 
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or float to set priorities 
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being completed on time, 
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paths that are critical, 


Manages buffers to 
minimize unplanned 
expediting, overtime and 
other costly deviations 
from schedule 
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= Task durations are relatively 
independent; Start/Finish dates 
are not 


= Task durations seldom are 
normally distributed which always 
presents analytic challenges 


= Charting the averages of 
averages tends to produce 
normally distributed data even 
where the underlying data are not 


= Large projects often have 
hundreds of tasks - presenting all 
the data points overwhelms the 
audience 
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8/7/2007 8/13/2007 a= 
8/13/2007 9/3/2007 —v 
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| September] October 
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‘are Requirements 15 days’ 
14.5 days? 


y it 21.75 days? 10/23/2007 — 

26 | Review functional specifications 1 day? G24/2007 p Developer 

27 | Identify modular/tiered design parameters 1 day? T Pa) sk my overerer 

28 Assign development staff lday? 9/26/2007 Developer — 

23. Develop code 1Sdays? 9/27/2007 | Durations p Developer 

30 Developer testing (primary debugging) 15 days? 10/2/2007 10/23/2007 "ug, Developer 
31 | [Development complete | Odays? 10/23/2007 10/23/2007 # 10/23 
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48 | Training 45.75 days? 9/24/2007 11/26/2007 S33 <$§£—T 
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74 E Deployment Sdays? 12/10/2007 =: 12/17/2007 = 


81 # Post Implementation Review 3days? 12/17/2007 = 12/20/2007 ss 


Milestones (Zero time 


and resource tasks that 
mark completion events) 
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Excel workbook’ 
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milestones? 
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Analyze the results of all verification activities. 
GQ(I)M Process and Indicator Supports: 
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Incorporate process-related work products, measures, and 
improvement information derived from planning and 
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GQ(I)M Process and Indicator Supports: 
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measures will be analyzed 
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GQ(I)M Process and Indicator Supports: 
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GQ(I)M Process and Indicator Supports: 
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Manage the project using the project plan, the other plans 
that affect the project, and the project’s defined process. 


GQ(I)M Process and Indicator Supports: 


A Documentation of approach to obtaining and analyzing 
the selected measures to manage the project and 
support the organizationG needs. 
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Establish and maintain definitions of the measures that 
are to be included in the organization’s process- 
performance analyses. 


GQ(I)M Process and Indicator Supports: 


A Selection of measures and definitions for appropriate 
insight into the organizationG quality and process 
performance 
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GQ(I)M Process and Indicator Supports: 


A Development of definitions of the measures and analytic 
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Select the defects and other problems for analysis. 
GQ(I)M Process and Indicator Supports: 


A Documentation of objectives established for 
measurement and analysis, specifying the measures and 
analyses to be performed, obtaining and analyzing 
measures, and reporting results 
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Establish and maintain the plans for deploying the 
selected process and technology improvements. 


GQ(I)M Process and Indicator Supports: 


A Establishment of measures and objectives for 
determining the value of each process and technology 
improvement with respect to the organizationG quality 
and process-performance objectives 
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Measure the effects of the deployed process and 
technology improvements. 


GQ(I)M Process and Indicator Supports: 


A Establishing objectives for measurement and analysis, 
specifying the measures and analyses to be performed, 
obtaining and analyzing measures, and reporting results. 
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Establish and maintain the plan for performing the process. 
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A Identification and documentation of measurement 
requirements to be included in the plan for performing the 
process 
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Monitor and control the process against the plan for 
performing the process and take appropriate corrective 
action. 

GQ(I)M Process and Indicator Supports: 


A Documentation of established measures for monitoring 
actual performance of the process. 
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results, and improvement information derived 
from planning and performing the process to 
support the future use and improvement of the 
organization’s processes and process assets. 


GQ(I)M Process and Indicator Supports: 


A Selection of appropriate measures to support 
future use and improvement of processes and 
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Stabilize the performance of one or more sub-processes to 
determine the ability of the process to achieve the 
established quantitative quality and process-performance 
objectives. 


GQ(I)M Process and Indicator Supports: 


A Selection of process and product measures to be 
incorporated into the organization® measurement 
repository to support process-performance analysis and 
future fact-based decision making 
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Ensure continuous improvement of the process in 
fulfilling the relevant business objectives of the 
organization. 


GQ(I)M Process and Indicator Supports: 


A \dentification of process improvements that would result 
in measurable improvements to process performance. 


All Others Bring Data 
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Seger Features 


8 The Guns Systems Software Engineering Program at BAE 
Systems wanted a way to reduce delivered operational defects 


Stabilize defect removal processes 

Provide measurable cost reduction 

Make data driven management decisions 

Provide early indication of delivered product quality 


Do De pe 
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i Features 


= CMMI Implementation 

= Good historical data set 

= Mature Project process set 

=" Defect removal processes were stable 
= Historical defect rate was available 
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BAE SYSTEMS 


Seed Features 


= Obtained S.W.E.E.P. (Software Error Estimation Program) 
= Based initial tests on David Card® Empirical Method 


ASee STSC Crosstalk March 2003 fManaging Software _ uality 
With Defects n 


= Started populating tool with the local data set 
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Features 


7 Use Historical Data to determine t-value and estimated defects 
»  _T-value is the point where the most defects are discovered 


7 Insert estimated defect values into tool 


=» Insert Actual defects discovered in a phase into tool at phase completion (replacing predicted 
values) 


= Determine if defect discoveries are within predicted bounds. 


If not, Perform Causal Analysis 
A Make process changes as needed 
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Seager Features 


= Construct Estimated Discovery Curve 
A100,000 lines of Code 
AHistorically we find 10 defects per KSLOC 
A1000 defects to be discovered 


= Determine number of phases 
A6 Phases (Req, DD, Code/UT, Int, SOT, FQT) 


= Estimated T-value 
ADefault T-value Is 2.21 


= Will result in a latent defect count of 2 defects 
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saeemeed Features 


"Number of defects discovered during defect removal 
reviews 


"Change Request type defects 


«Customer initiated changes 
AMay or may not result in a re-estimate 
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SVVEEP v4.1 - Example - Error Deterministic Quality Control Chart 


0.00 i i i it i ii 
Requirements CodingAJT Systems Quality Test 
Detailed Design Intigration Final Quality Test 
Phases 
Requirements Detailed Design CodingAJT Intigration Systems Quality Test (Final Quality Test 

(Lower 87.65 214.99 239.49 183.20 105.21 46.95 
i Nominal 

(LJ Upper 


Lower BE Nominal Upper 
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seemed Features 


=Nominal estimate would result in 52 defects at FOT 


= Set upper and lower bounds at 10% of Nominal 


= Will perform Causal Analysis if bounds are exceeded 
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SVVEEP v4.1 - Example - Error Discovery Data and Rayleigh Fitted Histograms 


300.00 
280.00 - 
260.00 
240.00 - 
220.00 - 
200.00 - 
180.00 - 
160.00 - 
140.00 - 
120.00 - 
100.00 - 
80.00 
60.00 ~ 
40.00 - 


20,00 - 


0.00 


Requirements Coding JT Systems Quality Test 
Detailed Design Intigration Final Quality Test 


Phases 


[Requirements Detailed Design CodingiT Intigration Systems Quality Test Final Quality Test 


‘DD Actual 


144.00} 0.00) 0.00 | 0.00) 0.00) 0.00| 


Oo Estimatec, 


97.00 230.00 257.00 195.00) 108.00) 43.00) 


Actual 


MB Estimated 
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sedermie- i) features 


= 144 discovered defects 

=Exceeded 10% Tolerance 

=Perform Causal Analysis to determine why 
Alncomplete Requirements? 
ABetter Discovery Rate? 
APoorly written work product? 


«Finding more defects earlier results in a drop in delivered 
defects to FQOT (43 defects instead of 52) 

= Management makes decision to continue with 
development but with more time spent on defect removal 
reviews 
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300.00 -— 


280.00 - 
260.00 ~ 
240.00 - 
220.00 - 
200.00 

180.00 - 
160.00 


140.00 - 


Errors 


120.00 - 
100.00 - 
80.00 - 
60.00 - 
40.00 - 


20.00 - 


0.00 


SVVEEP v4.1 - Example - Error Discovery Data and Rayleigh Fitted Histograms 


’ 7 5 ' t . 


Requirements CodingAJT Systems Quality Test 
Detailed Design Intigration 
Phases 


‘Actual 
_ Estimatec 


Requirements Detailed Design CodingAT \Intigration Systems Quality Test Final Quality Test 


144.00 270.00) 0.00) 0.00 0.00 | 0.00) 
| | | t | 
97.00 | 234.60) 253.85| 191.85) 104.85] 39.85 


Actual 


Wl Estimated 
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Pewee Features 


= Discovered 270 defects 
"Perform Analysis to determine: 
ACause of defects 
AQuality of work product 
Almpact to defect latency 
Almpact of more review time 
=Latency Rate is now 39 defects 
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SVVEEP v4.1 - Example - Error Discovery Data and Rayleigh Fitted Histograms 
300.00 
280.00 - 
260.00 - 
240.00 
220.00 - 
200.00 - 
180.00 - 
160.00 - 
2 
2 140.00 
wi 
120,00 - 
100.00 - 
80.00 - 
60.00 - 
40.00 - 
20.00 - 
0.00 ° : : : : ' 
Requirements CodingAJT Systems Quality Test 
Detailed Design Intigration Final Quality Test 
Phases 
[Requirements [Detailed Design [Coding UT Intigration ‘systems Quality Test, Final Quality Test | 
OD Actual 144) 00) 270.00 0.00 0. 00 | 0. 00] 
i Estimatec 97.00 | 234.60 | 195.1 5| | 1 07 .45| 43.1 5| 
Actual Estimated 
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seemed Features 


= Discovered 243 defects 


#=Within Bounds of Estimate 


«Latency rate is 43 defects 


= Management decides to continue with no change to 
process 
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SVYEEP v4.1 - Example - Error Discovery Data and Rayleigh Fitted Histograms 
300.00 
280.00 - 
260.00 - 
240.00 - 
220.00 - 
200.00 - 
180.00 - 
160.00 - 
2 
© 140.00- 
rT 
120.00 - 
100.00 - 
80.00 - 
60.00 - 
40.00 - 
0.00 ¢ 
Requirements CodingAJT Systems Quality Test 
Detailed Design Intigration Final Quality Test 
Phases 
‘Requirements Detailed Design CodingJT Intigration ‘systems Queallty Test! Final Quality Test 
DO Actual 144.00, 270.00 243.00] 200.00 ~ 0.00] O o0| 
i Estimatec | 97.00 | 234.60 253.85 195.15 40.75 
Actual Estimated 
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feed Features 


= Discovered 200 defects 


=Within estimated bounds 


«Latency rate is 40.75 defects 


= Management decides Work Product development is 
performing the way they want it to. 


"No Changes to Development Process 
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SVYEEP v4.1 - Example - Error Discovery Data and Rayleigh Fitted Histograms 
300.00 
280.00 - 
260.00 - 
240.00 - 
220.00 - 
200.00 - 
180.00 - 
160.00 - 
2 
© 140.00- 
rT 
120.00 - 
100.00 - 
80.00 - 
60.00 - 
40.00 - 
0.00 ¢ 
Requirements CodingAJT Systems Quality Test 
Detailed Design Intigration Final Quality Test 
Phases 
‘Requirements Detailed Design CodingJT Intigration ‘systems Quality Test) Final Quality Test / 
DaActual 144.00, 270.00 243.00/ 200.00 “105.00 
i Estimatec | 97.00 | 234.60 253.85 195.15 40.75 
Actual Estimated 
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Seemed Features 


= 105 Discovered Defects 


=Within estimated bounds 


«Latency rate is 40.75 defects 


= Management decides Work Product development is 
performing the way they want it to. 
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SVYEEP v4.1 - Example - Error Discovery Data and Rayleigh Fitted Histograms 


Errors 


Requirements CodingAJT Systems Quality Test 


Detailed Design Intigration Final Quality Test 


Phases 


| Requirements 


oO Actual 
|| Estimatec | 
Actual 


__ Detailed Design 
270, oo 
234.60 | 


CodingiUT Intigration 


200 oo} 


253.85) 195.15] 105.05 | 


EB Estimated 
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seemed Features 


= 39 Defects discovered 
= Project delivers under it® goal of 52. 


=Product Quality Objectives have been met and Work 
Product can be passed to customer 
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Seager Features 


= By performing more intensive reviews at the start of the 
development cycle, better work product quality was 
obtained. 


= The chance to perform Defect Profile Analysis will provide 
information on which processes injected the most defects 


AOID activity 
ACausal Analysis cycle 
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Syeemieed Features 


"By performing analysis on discovered defects during 
development, one can impact the number of delivered 
defects to FQT. 
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eed Features 


= This analysis works quite well for waterfall type projects 


= Have witnessed zero delivered defects for several delivery 
cycles 


=Provides value added measurement data 


=Covers a whole host of CMMI evidentiary requirements 
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CMMI SCAMPI Appraisals 


The People The Process The Results 


Robin B. urst 
United Space Alliance, LLC 


robin.b.hurst_usa-spaceops.com 
(2) -~- 2 


th Annual CMMI Technology Conference and Users Group 
Track 


Copyright © 2007 by United Space Alliance, LLC. USA 
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Your an 


Awho are the participants 
Awhat activities occur 
Awhat artifacts are produced 


AWhat actions reactions should result 


() USA best practices or helpful hints 
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A PIID developers (Practice Implementation Indicator Description) 
A Lead appraiser 

Appraisal team members 
A Interviewees (project team members) 


A Other Participants 
| SEPG (Software Engineering Process Group) 
| Project managers 
| Process architect 
| Project team members 


A All USA elements had these participants involved (some known by 
other titles or names) 
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i Sponsor — Identified as the person responsible for all appraisal 
activities. Allocates and approves all schedules, personnel and 
seid involved in the activities before, during and after the 
appraisal. 


A soft Skills 


1 Able to focus on important goals and objective 
1 Good cheerleader encourager 
| Provider of resources (not just words but deeds too) 


Communicates — gives and receives information 
| PIID developers 
| Lead appraiser 
| Project team members (interviewees) 
| Process architect(s) 
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N Prepares PIIDs for the selected projects to be appraised 


A ldeally not more than - people 
1 Group size dependent on organizational size () 
A Soft Skills 
1 Familiar with CMMI (Capability Maturity Model Integration) 
| Familiar with the projects 
| Understand the organization’s processes 
1 Can work alone or as a team member 
[ 
; 
‘ 


Possess good written communication skills 
Productive (able to work on time constraint schedules) 
Provide attention to detail 
A. communicates — gives and receives information 
1 Sponsor 
| Lead appraiser 
| Project team members 
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- Authorized SCAMPI (Standard CMMI Appraisal Method for 
Process Improvement) Lead Appraiser contracted by 
organization to conduct SCAMPI appraisal activities 


May consult with organization prior to appraisal activities 
| Reviewing PIIDs 
| Reviewing process documentation 


A sott Skills 
| Possess good oral and written communication skills 
| Facilitative 
|! Knowledgeable of industry and CMMI best and acceptable practices 
| Creative 
Communicates — gives and receives information 
| Sponsor 
| PIID developers 
| Appraisal team members 
| Process architect(s) 
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‘\ Trained team members — per Method Description Document 
A ideally 

| team members and _ lead appraiser (minimum _) 

| internal organization members 

1 external members 

External organizations or companies 

A Soft Skills 

| Possess good oral and written communication skills 

1! Can work alone and as a team member 

1 Can maintain objectivity and confidentiality 

| Good time manager 

Communicates — gives and receives information 
| Sponsor 
| PIID developers 


| Lead appraiser 
| Process architect(s) () 
A Some appraisal team members may also be PIID developers reviewers. 
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joants - Interviewees 


‘\ Interviewees — Project team members who represent process area 
activities (SEPG, project management, CM, S A, SW engineering, 
testing, etc). During interviews, activities annotated in PIIDs are 
expected to be affirmed by those being interviewed. 


A May provide artifacts to PIID developers as requested 
A Usually have access to PIIDs prior to appraisal activities 


A Soft Skills 
| Possess good oral communication skills 
| Knowledgeable of the project activities and processes 
1 Champion of the process (if possible) 


Communicates — gives and receives information 
| PIID developers 


| Lead appraiser 
Best practice — “Charm School” for interviewees () 
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pant - Communications 


PIID Developers 


Process Architect 
SEPG 


Interviewees 


(Proj. Team Members) Appraisal Team 


Communication its essential between all involved 


participants in appraisal activities. 
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on 


ties — Before, During and After 
a Before the appraisal 

Apburing the appraisal 

Aafter the appraisal 
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Before the appraisal 


Determination of appraisal needs (analyze requirements) 
Selection of lead appraiser 


Develop appraisal plan 
Identification of appraisal scope 
Projects to be appraised 
A Identification of interviews to be held and interviewees 
Identification of appraisal team members 
Risk management 
A scheduling on appraisal activities (including interviews) 


Training of appraisal team members 
Identification training of PIID developers 
PIID annotation () 

Consultation 

PIID population 
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A USA determined the following reasons for being appraised 
against the CMMI 


1 New business objective 


| Maintenance of existing CMMI rating 


| Meeting a customer expectation 
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___.._|}+ Lead Appraiser Selection 
‘\ Determine through interviews and evaluations the following 
Aexperience () 
A availability 
Expectations 
Model interpretations 
Consultations 
Resources 


Ability to communicate 
Ability to support your needs and objectives 


Athe lead appraiser must be able to work and communicate 
with the sponsor and PIID lead. Final lead appraiser 
selection was determined by the sponsor. 


” USA 
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* ._ D/Train Appraisal Members‘ 


‘\ Method Description Document (MDD) identifies the training and 
experience base needed to comprise a qualified appraisal team 
(total engineering and management team experience) 


Good team makeup 
1 the team have previous appraisal experience (4) 
| New teams have some type of team facilitation training 
Belbin, Meyers Briggs 


A May be asked to help interpret practices for the PIID developers 
while PIID annotation is being performed. May contact the lead 
appraiser for interpretational guidance 


USA determined that it was good to always train backup appraisal 
members for a given team to mitigate risks for retraining due to 
team member loss or unavailability (4) 
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= BG Features D/Train A JK raisal Members 
‘\ Organized into mini-teams 


I i een four 2-member mini-teams 
One member from the organizational unit and one external 


member 
APreferably one experienced team member per mini-team () 
| Assigned a group of process areas to review 


1 One Example 
Organizational PAs - OPF, OPD IPPD, IPM IPPD, OT 
Support PAs—PP A, CM, MA, DAR 
Engineering PAS— RE M,RD, TS, PI, ER, AL 
Management PAs — PP, PMC, RSKM 


® 
2 USA 
Filename Rage ee ee eee 


United Space Alliance 


Your complimentary i 


use period has ended. 


Ecomplete — accomseta 
Ee ane Features / D/ Tr. a i nn P / / D D eG vel O 


‘\ Select a lead to coordinate and plan PIID preparation and team 
activities 


| Possibly an appraisal team member — must be knowledgeable of CMMI 
practices and appraisal methods (some danger of objectivity in the appraisal ) 


Aselect PIID team members -_ or 2 from each identified 
organizational unit to be appraised 


1 Primary and backup always works to your advantage 
1 Support (SEPG) member used to populate organizational PAs (OPF, OPD, OT...) 


Ensure that line management understands the impact (hours) of 
PIID team members involvement in PIID preparation 


1 Communicate commitment of resources (time, facilities) to all relevant 
stakeholders (line management, sponsor, facilities, team members, etc) 


| Create team charter cee action team (PAT) or similar) with appropriate 
signatures acknowledging commitments, resources and manage as a project 
Atrain PIID developers in 


1 CMMI model 
1 PIID tool of choice 
1 Team operating rule and expectations (contained in charter) 
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: This activity involves documenting what the practices of the 
CMMI model means in the PIID 


A Mapping the projects’ organization’s processes to the model 
practices and providing evidence of implementation of the 
practices 


| Generic statement describing what artifacts should be provided for 
each practice and what makes it appropriate 


A Labor intensive task for a team who has never annotated PIIDs before 


A Example PIID with annotation (no project or org. attribution) on 
next slide 


| Green italic text explains how the artifact meets the intent of 
the practice 
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Project]M ]Model Evidence Interview | Char| OU 
Link Link 


Estimates of project 
planning parameters are 
established and 
maintained. 


SP 1.1]Establish a top- 


level work Pl Project1 Project Plan P1 Project Plan Document # 
breakdown Document # xxxxx (section je |xxxxx Approval/ Concurrence 
structure (WBS) y.y, Z.Z, revision history) page 
to estimate the tml 
scope of the Statement of Work, Project Approval and concurrence 
project. specific WBS identified, pages from both the SEPG 
revisions to plan due to and the relevant 
template changes (including stakeholders for the project 
WBS) 


P2 Project2 Project Plan P1 Project Plan Document # 
Document # xxxxx (section je |xxxxx Approval/ Concurrence |the_pp2 
y.y, Z.Z, revision history) page approval.h 
Statement of Work, Project Approval and concurrence 
specific WBS identified, pages from both the SEPG 
revisions to plan due to and the relevant 
template changes (including stakeholders for the project 
WBS) 
(ea Es [a (a ee) een | OST (Pr PD (RE ew] 
estimates of the |P2 
attributes of the 
work products 
and tasks 
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A statistics — Typical CMMI Maturity Level Appraisal (not 
including Supplier Agreement Management) 


1 Process Areas (PAs) 
Process Management PAs 
Project Management PAs 
Engineering PAs 


Support PAs 
1 2 Specific Practices (SPs) 
Approximately pieces of evidence 


1 2 _ Generic Practices (GPs) 
Approximately pieces of evidence 
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~~. Lead Appraiser Consultation 
‘\ Types of consultation 
| Process document reviews 
1 PID annotation reviews 
| Information gathering 


Benefits 
| Identify gaps before process institutionalization or identification in 
appraisal settings 


1 Consistent interpretation from lead appraiser 


A Drawbacks 
| Expense and schedule 


| Academic interpretation vs real world implementation due to lack of 
understanding of the organization’s context 
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A Identify “focus” and “non-focus” projects 


A Identify PIID artifacts which must be obtained from the selected 
projects to complete PIID population (per annotation) 


Create a data management plan for controlling the PIID artifacts 
which will be used for the SCAMPI appraisal 


A USA used the technique of “Relocation of Artifacts” 


| Reduced security impacts for appraisal team 


| Reduced risk of content change or archival 


| Used tool snapshots 
No learning curve for team on tools 
Current content capture 
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Pre Class B 
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PIID Effort 
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Filename 
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60% 
50% 
40% 
30% 
20% 
10% 

0% 


PIID Effort Average Distribution 


Pre Class B Class B- RR RR - SCAMPIA 
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A During the appraisal 
| Prepare participants 
1 Review objective evidence (PIID review) 
1! Conduct interviews 


| “Tag” notes and PIIDs 


Identify weaknesses of the practices 


erify implementation of practices — characterize the extent to 
which the practices meet the model 


Review with team to reach consensus 


Prepare and conduct preliminary findings presentation 


Generate and present appraisal results 
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emeed Features 


‘\ In addition to the opening briefing performed by the lead 
appraiser, USA has found great benefit in having the home 
organization provide an overview presentation to help the 
appraisal team level-set the process and tools that the 
organization is using to meet the model expectations 


| This presentation is typically presented by the software project 
manager or process architect and includes 
Organizational structure and appraisal scope 


Information about the organizations standard software process 
(including a process map) 


Description of process assets and access methods (web 
interface, etc) 


Aorganizational approach toS ACM ER ALIPPD 
Size and nature of project being appraised 
Glossary of organizational unit or project specific terms 
Demonstration of tools used by the project organization 
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" —_ +Review Objective Evidence’ 


‘. PIID review... document review...artifact review... 


A Factors that play an important role in determining how to 
review a process area (PA) 


| Appraisal experience 
| The PA being reviewed 
1 The organizational unit — level of commonality 
| Readiness of the PIIDs 
Remember. . . 
A 2 Specific Practices (SPs) 
| Approximately pieces of evidence 
A 2 Generic Practices (GPs) 
| Approximately pieces of evidence 
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| meg Features evi Cw Ob ecti ve E vi den ce’ 


i epeiaiee team members (mini-team members) focus 


| Organizational processes and standards comply with model 
expectations 


Projects implementing model per organizational processes and 
standards 


Note area of concern — possible question 
Look for threads 


(NON-SCAMPI A) identify and document PIID issues 
Evidence does not support the practice 


A Each team member has a strength — use it during objective 
evidence review 
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. = - Conduct Interviews’ 


A For CMMI activities lead appraiser does all interviews — asks all 
questions 
1 Mini-team members take notes 


| Mini-team members provide specific questions needed to support 
evidence provided in PIIDs (affirmation of the practice) 


| List of questions provided to mini-teams for following during 


session () 


A Example of interview question spreadsheet on next slide 
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SG1 

Estimates of SP1.1 

project planning |Establish a top-level work 
parameters are |breakdown structure (WBS) 
established and jto estimate the scope of the 
maintained. project. 


SP1.2 

Establish and maintain 
estimates of the attributes of 
the work products and tasks 


SP 1.3 

Define the project lifecycle 
phases upon which to scope 
the planning effort. 

SP 1.4 

Estimate the project effort 
and cost for the work 
products and tasks based 
on estimation rationale. 
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- Conduct Interviews2 


SPMs 


PROJECT PLANNING (PP) 


Where and how is the 
framework to plan, organize 
and control the work done on 
the project documented? Do 
you have a WBS? Where is it 
documented? 

What parameters do you use to 
estimate work products and 
tasks (document size, critical 
computer resources, and 
source size, bandwidth)? 

How do you determine the 
project's lifecycle phases so 
you can scope the planning of 
the project? Where is _ the life 
cycle documented? 


How do you estimate the 
project's effort and costs? What 
rational is used? 
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Atag notes and PIIDs per the appraisal methodology 


A Document weaknesses in the PIIDs for projects which do not 
have artifacts and affirmations which support the intended 
implementation of the model practice 


| Joint agreement by mini-team on weaknesses 
| Suggested improvements may also be documented (I O) 
| PIID feedback may also be included (non-SCAMPI A) 


A Many hours have been wasted in wordsmithing during an 
appraisal — don’t fall victim of this trap 


1! ome organization should take lead in “crafting” words for 
findings so that projects and improvement opportunities are 


easily understood (d) 
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eview to Reach Consensus 


A erify implementation per the appraisal methodology 


A Reaching consensus can be the hardest (or easiest) part of the 
appraisal 
| Deciding factors 


ow well the artifacts were reviewed and mini-team notes were 
taken 


Mini-team consensus and objectivity 
1 Leaving “I, we, me at the door” 
ow weaknesses are written and explained 


A Each PA is reviewed with the entire appraisal team 


| Each mini-team presents their insight and documented weaknesses 
for the PAs assigned to them 


1 uestion comments from entire appraisal team 
APick your battles 
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»  +fepare Findings Presentations 


‘\ Weaknesses from PIIDs are gathered “polished” into “finding” 
and slides are produced 
1 Global weaknesses are documented 
| Practice weaknesses are documented 
| Process improvements are documented 


A Documenting weaknesses... 
1 Use “some”, “most” , not “all” when speaking of projects 
| Don’t reiterate the model practice 
1 Use words that have meaning to the organization 
1 Only identify the weakness once (global) 


A Non-attribution of appraisal output to individuals ARC 2 
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_ erate and Present Appraisal Results 


A Derive final finding using preliminary findings statements 


| Rewording preliminary findings to make them more 
understandable to the organization may be needed 


A Rate each specific goal and generic goal within the scope of the 
appraisal and obtain consensus with the appraisal team 


Many organizations conduct executive final findings briefings 
prior to the organizational briefing 


| Ensure appraisal sponsor has sound understanding of the results 
and provides them with an opportunity to decide what they are 
going to say to the organization about the appraisal results () 
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A after the appraisal 
| Archival of appraisal assets 
| Appraisal team lessons learned 


| Lead appraiser provides reports to the SEI 
Appraisal plan 
Final findings presentation 
Appraisal disclosure statement 


Appraisal evaluation forms (lead appraiser, sponsor, appraisal 
team members) 
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'_ Archival of Appraisal Assets 
A Dispose of sensitive material and preserve important data and 
records from the appraisal 


A Gather appraisal artifacts and data for the sponsoring 
organization 


1 Typically — opening briefing, cleansed PIIDs, preliminary findings 
and final findings briefings, appraisal disclosure statement (ADS) 
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— Lessons Learned &) 


A conducted with the lead appraiser and the appraisal team 
1 Identify best practices that occurred during the appraisal 


| Identify areas of the appraisal that need to be improved or that did 
not work and do not need to be repeated 


A. Experienced appraisal team member keep a running log of 
these items during the appraisal 


A Need to be taken seriously and conducted on site 
A Example 


| Best practice - Mini-team members were well paired (previous 
appraisal experience no previous appraisal experience) in their area 
of expertise (engineering, process, project management, support) 


Needs improving — Process areas were not well divided among the 
mini-teams. When mini-teams completed their area assigned library 
hours were not maintained. 
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_  aisals Over — What Now? 


A sponsor SPM meets with internal organizational appraisal team 
members 


| Understand the findings 
1 Review PIIDs 


1 Request corrective action plan 


A Internal appraisal team members should be prepared to explain findings to 
the sponsor 


Possibly extent of corrective action necessary 


Magnitude of impact to the organization 


Should always maintain non-attribution to ensure detailed information 
about individuals does not lead to inappropriate consequences () 


following the appraisal 
A All meetings and feedback should be positive and reflect a job well done 


Sponsor (and possibly the appraisal team) meets with the lead appraiser 
to ask questions and review their corrective action plans 
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A It take a lot of hard work and dedicated people to have a 
successful appraisal (sponsor, PIID developers, appraisal team 
members, lead appraiser, interviewees, ...) 


A Many hours are spent by many individuals in the preparation, 
execution and follow up work of an appraisal 


Ateam work is the key to success 
Aa positive attitude and a sense of humor will help you get there 
A No two appraisals are exactly the same 


Be willing to listen, be willing to change, be willing to admit you 
could be wrong 


A Pick your battles 
A Be flexible 


A ou will have findings No one is perfect 
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Robin’s quote — | don’t have to fix it in my world, | just have to 
document it in yours. 
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Charleston A Large Systems Software Engineering Organization 
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Science: 
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engineering or 
computer-related 
discipline 


e—Contracts Supply, 2 


Finance Budget, 


General Clerical, 


Program Mgnt, Logistics, 


AThe solutions to the global war on terror developed by SPAWAR 
result from good systems and software engineering 


ASystems engineering is our core competency 
ATotal workforce of 2, employees 
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A ision 

1 Develop and maintain a World Class Systems Engineering Organization 
AApproach 

1 Achieve Command-wide operational consistency 

1 Based on ISO 152881 systems engineering 

1 Based on ISO 122071 software engineering 

1 Measure using best practices of CMMI® 
AGoals 

1 CMMI Maturity Level 2 by April, 2005 

1 CMMI Maturity Level 3 by April, 2007 
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st SPAWAR Systems Center to Achieve ML2 and ML 
New Goal Maturity Level by 2 
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Command-wide Policy 


CRITICAL SUCCESS FACTORS FOR CMMI 
(Create vision that is urgent) 


Critical Success Factors 


IMPLEMENTATION 
Strategy and Plan (Include 
knowledge of why change is 


necessary and benefits) 


Assign Responsibilities 
(Strong Change Agents are essential) 


Senior Management Support 


Provide Training 
Provide Resources and Funding 
(New Organizational Structure 


Usually Needed) 


Build a Central Repository 


Measure and Communicate Progress 
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APrior to 2 
1 Code 70 had experience with SW-CMM® 


2 


1 SSC-C Process Improvement (PI) effort began 


| Code 70 developed PI Policy for SE, SW, and Security 
Engineering using SEI CMM® and CMMI® 


I ae 70 Engineering Process Group expanded to Command- 
wide 


| Engineering Process Office (EPO) Website started 
1 Pilot Projects selected and evaluated 
Some templates published 
A2 2 
| Began developing and delivering training 
Began conducting Class FCOassessments as progress checks 
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2 


1 Established and Funded Dir. of Engineering Operations position 
AStaffed Engineering Process Office (EPO) 
1 Developed Organizational Standard Policies 
APolicy for each CMMI® Level 2 and 3 Process Area 
| Developed Organizational Standard Process Manuals 
ATop Level 
A Systems Engineering 
A Software Development 
A Software Maintenance 
ASupporting Processes 
A Process Manual for each CMMI® Level 2 and 3 Process Area 
Developed plan templates 
Coached and mentored pilot projects 
Built tools 
Developed and delivered training 
Performed interim assessments 
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Systems Center 
Charleston 


2 


1 Conducted project-level Maturity Level (ML) 2 SCAMPIS™ Class 
mMoappraisals 
A6 Projects Appraised 
AG Achieved ML2 


AApril 2 


1 Conducted Command-level ML2 SCAMPIS™ Class MAdappraisal tT 
First SPAWAR Systems Center to achieve Command-level ML2 


World Class 
Systems 
Engineering 


Systems 


Capability 
Center Maturity 
Charleston Model 
Integration 
(CMMI®) 


MATURITY LEVEL 
April 28, 2005 
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Systems Center 
Charleston 


AAddressed the three Organizational Process Areas 
early to provide a smoother transition to ML 
1 igahizational Process Focus (OPF) - Purpose: Plan, implement, 
and deploy organizational process improvements based on an 
understanding of the current strengths and weaknesses. 
ADetermined Process Improvement Opportunities 
1 Management commitment! the PI strategy 
1! Benchmarked current state, addressed identified needs/gaps 


APlanned and Implemented Process Improvements 


| Determined Scope, Model (CMMI-SE/SW), Approach (Staged, but 
appraise using Continuous) 


1! Created appropriate teams to champion PI efforts 


ADeployed Organizational Process Assets and Incorporated Lessons 
earne 


| Shared sample project plans, improvements, etc., across the 
organization 


== sEIPartner 


‘WecHsor 


16 Approved for public release; distribution is unlimited (18 OCT 2007) 


Your complimentary 
use period has ended. 
Thank you for using 
PDF Complete. 


he Second Wave — ML2 to ML 2 


seemed Features 


Systems Center 
Charleston 


AAddressed the three Organizational Process Areas 
early to provide a smoother transition to ML (con’t) 


| Organizational Process Definition (OPD) 1 Purpose: Establish 
and maintain a usable set of organizational process assets and 
work environment standards. 


ADeveloped EPO website, which is a repository for standard process 
manuals, SOPs, checklists, etc. The site also contains Tailoring 
criteria and other useful resources such as sample plans, etc., 
shared with the SSC-C organization by its projects 


ABuilt SSC-C eA fedleaels ls Measurement Repository (OMR) for 
projects to use for ae their projects and capturing 
standardized cost, schedule, and process performance 
measurement data 


1! Defined Balanced Scorecard measures directly related to CMMI® 
and Process Improvement 
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Systems Center 
Charleston 


AAddressed the three Organizational Process Areas 
early to provide a smoother transition to ML (con’t) 


| Organizational Training (OT) - Purpose: Develop the skills and 


knowledge of people so they can perform their roles effectively 
and efficiently. 


Aldentified the training needed by the organization 
AObtained and provided training to address those needs 
AEstablished and maintained training capability 
AEstablished and maintained training records 
AAssessed training effectiveness 


| Objective evaluation of OT process performed by the Process and 
Product Quality Integrated Product Team (PPQA IPT) 
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ASSC-C organization developed Tailoring Guidelines 
ASSC-C Projects developed ML2-to-ML3 Action Plans 


ADeveloped internal fself-assessmentdprocess for measuring 
ongoing implementation of ML2 processes 


AContinued enhancing ePlan Builder tool to create new plans 
(e.g., SEP/SEMP) that are ML3 compliant 


A Updated/Improved existing plans 
AProvided additional CMMI® Training 


AAdded Work Breakdown Structure Tool and Architecture 
Development Web-Based Training Course 


AContinued to Measure and Communicate Progress 
AMaintained Momentum and Commitment to Goals 


Tr ===} SElPartner 
19 Approved for public release; distribution is unlimited (18 OCT 2007) Technical Sofiware Serviess,:!nc: 


Your complimentary i _ 
use period has ended. 


yy PDF Thank you for usin a | = | 
Complete" p2etrere Timeline 2 = -2 


seem) features 


Systems Center 
Charleston 


AMay — Dec 2 


| Updated Organizational processes with ML3 language 


| Built ede ail pelea Measurement Repository (OMR) to track 
cost, schedule, and process performance measurement data 


| Developed Sample ML3 plans 


| Projects: Built ML2 to ML3 transition plans 
ACoaching and mentoring continued 


| Conducted project-level Maturity Level 3 SCAMPIS” Class MAO 
appraisals 


A6 Projects Appraised between June and December 
A5 Achieved ML3 


| Projects worked to correct consistent weaknesses in Peer 
Reviews, Decision Analysis and Resolution (DAR), PPQA 
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AJanuary 2 

1 1 additional project achieved ML3 

t Collected data from 30+ mon-focusedoprojects 
ATailoring Guidelines 
AProject Management Plans 
ASEMP/SDPs 
APPQA Plans 
ACM Plans 
AM&A Plans 


AFebruary 2 
| Conducted 5-day Readiness Review 
1 Collected additional artifacts needed 
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AApril 2 


1 Conducted Command-level ML3 SCAMPIS™ Class Moappraisal 1 
First SPAWAR Systems Center to achieve Command-level ML3 


1 9 Projects in appraisal scope | 7 Focused, 2 Non-Focused 
A>8000 artifacts submitted, 164 interviewees 

| SEI Senior Member was Lead Appraiser (Team Leader) 

1 2 other SEI Authorized Leads on the Team 

1 1 Government person from NSA 

1 1 Government person from SSC-C 

| 3 Appraisal experienced team members 
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Systems Center 
Charleston 


ACarefully select Initial Projects 
1 Start with interested projects 
AHigh Sponsor interest 
AStrong need/desire to improve 


ASet Guidelines (criteria) that yield benefits, for 
example, SSC-C’s CMMI Projects meet the following 

1 Systems or software engineering effort 

| Funding directly with SSC-C 

| SSC-C performs the Project Management function 

| SSC-C PM is directly responsible for product delivery 

1 Multi-year effort 

| Over $2M per year 

| Not limited to level of effort for services 

| Not merely a pass-through contract 
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Systems Center 
Charleston 


AAssign a CMMI resource to each project 
1 Strong facilitator with strong CMMI® knowledge 


1 Conduct regular (at least monthly) process-focused meetings to 
ensure steady progress 


Alnclude all key process area members (including contractors) 
| Review projectS plans, SOPs, work products 


| Explain process area practices to the team6 subject matter 
experts 


ARelates model to project 

AHelps team define typical work products 

AHelps team identify and collect direct and indirect evidence 
1 Conduct mini assessments to benchmark progress 
| Share/provide organizational tools, templates 
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Systems Center 
Charleston 


AProject Team 
1 Project Manager - involved and committed to success 


1 Document specialist/Technical Writer role for coordinating 
documentation, revisions 


| Active, skilled PPQA manager is a great benefit 
AAlso can serve as the Measurement Analyst 


| Useful plans are built by the key players; shelfware is built by the 
novice or new contractor 


1 Don@ let one person wear too many hats 
AResource the team properly 
1 New technology and complex systems are NOT necessary for 
SUCCeSS 


AA Customer that supports the initiative is a plus 


“WecHsort artner 


e Services, Inc. 


25 Approved for public release; distribution is unlimited (18 OCT 2007) 


Your complimentary : - — a 
use period has ended. 


mgcompiete xen te@Ss Factors of Implementation 


sewer ty Features 


Systems Center 
Charleston 


ARecognize and Publicize Early Successes 


| @®roject-levelOSCAMPIs provided early successes due to 
conducting the appraisal using the reontinuous representationoof 
the model 
A Scope of appraisal looked at all 7 ML2 PAs, then 11 ML3 PAs 
Alf all the PAs were satisfied, then the project achieved ML2 and/or ML3 through 
equivalent staging 
AOr, Projects received Capability Level 2/3 for various PAs satisfied (e.g., CM, 
SAM, REQM, PP, PMC, TS, Pi, DAR) 
| Led to BIG success! - SSC-C became the first SPAWAR Systems 
Center to achieve CMMI® Maturity Level 2 (April 2005) 


| Continued similar approach to Maturity Level 3 


A 1st Successful ML3 Program i July 2006 
A4 more projects achieved ML3 in late 2006 


| Command CMMI® Maturity Level 31 April, 2007 
A 1st SPAWAR Systems Center to achieve ML3 


===} SElPartner 


‘Wecusort 


26 Approved for public release; distribution is unlimited (18 OCT 2007) eal Sothware Ser 


eg Features 


Systems Center 
Charleston 


Appraisal Planning Execution 


Measuring Progress 
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A SEI staff members were involved in the SSC-C 
Class “A” SCAMPIs 


ARequired early planning to get each SEI staff 
member’s commitment to appraisal dates 


ABuilt detailed schedule for ML2 and ML _ project and 
organizational-level appraisals 


AObtained commitment from project team members 
concerning availability on appraisal dates 


AReserved conference and meeting rooms well in 
advance 
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APre-Readiness Reviews (PRRs) helped to ensure 
projects were ready and the Formal RR would 
lead to - coverage 


| Used Appraisal tool to conduct PRRs 


AProvided early and easy access to the direct and indirect 
evidence for each process areaG specific and generic practices 


AProvided means for communicating appraisal team comments 
1 Used convention to denote status of each practice 
(e.g., PRR-SG: Direct OE satisfies practice OR 
PRR-SG: Direct and indirect OE is too old) 
AProvided early feedback to the projects 
AProvided easy upload of new artifacts supplied by projects 
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AFormal RRs conducted on-site with Appraisal 
Team Members (ATMs) 


| SEI Lead Appraiser and ATMs worked as a team 


| Used Appraisal tool to conduct RR 


AProvided easy access to the direct and indirect evidence for 
each process areaG specific and generic practices 


AProvided means for communicating appraisal team comments 
1 Used convention to denote status of each practice 


(e.g. RR-CS: Direct OE indicates performance of practice 
OR RR-CS: Direct and indirect OE is too old) 


AProvided good feedback to the projects on items still missing 
AProvided easy upload of new artifacts supplied by projects 
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Systems Center 
Charleston 


ASCAMPI®" Class A appraisals conducted on-site 


1 Involved mostly the rinterviewoprocess since RR 
ensured direct and indirect coverage was evident 
| Used Appraisal tool to conduct SCAMPIS™ 


AAffirmation section of tool allowed for easy update following 
each interview 


ATool allowed primary team member to select practice 
compliance and secondary member to concur (or not) 


AAuthorized lead appraiser (team lead) then verified each 
practice within the process area 


ABuilt-in color coding provided easy visibility to fweaknessesO 
AFacilitated voting process at Goal level and Process Area 
| Each project-level ML3 SCAMPIS™ conducted in 5 


days and Command-level ML3 SCAMPIS™ conducted 
in 10 days 
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ASenior Management support is critical to success 
ATraining 
i Everyone needs to be engaged 1 firain the masseso 
1 Specific training for process owners/subject matter experts 
AUtilize Teams (IPTs) as champions of specific processes 
1 Multi-department representation 
1 Change agent mentality 
1 Process-focused charters 
AResource Properly 


1 Implement with projects that want to improve, can benefit from efforts, 
and that recognize own weaknesses 


1 EPO staff provided skilled coaching, resources, support, and tools 
1 Project members learned by doing and maintaining 
AGoals and Publicity 
1 Keep goals to sizable bites (projects) 
1 Publicize successes; Share best practices 
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AProvide CMMI mentoring and coaching for projects 
selected for an appraisal 


ABuild detailed schedules for appraisals early in 
planning phase to use as a roadmap 


APlan early in order to obtain project team member and 
appraisal team member commitment to appraisal 
dates 
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fed Features 


Systems Center 
Charleston 


Alnvest in an Appraisal Tool to facilitate easy 
collection and evaluation of appraisal data 


APerform a Pre-Readiness Review to ensure minimal 
Selag gaps are identified at the formal Readiness 
eview 


AConduct individual project appraisals to ensure 
successful organizational appraisals 


ADocument Lessons Learned from conducting 
appraisals to improve the appraisal process 
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ABusiness Results 
t SCN: Alhey see us as a model and want to increase our efforts.o 


1 Automation Program: rWe had hundreds of sites and there was a 
need for a structured organization to put a ovrapperoaround that 
and control it. CMMI became the wrapper.o 

t CICS: FCMMI was key to achieving the project goal.o 


1 VIDS: AThe VIDS failure (2000) motivated implementing CMMI 
because the team needed to change course or the customer 
would have no confidence in system development. It was a 
tremendous successe oO 


AOthers Asking for elp 
1 PMS 4081 CREW program 
| SESG/ NAVAIR / NAVSEA 
| Marine Corp 1 Quantico 
1 Air Armament Center, Eglin AFB 
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Beyond Maturity Level 


Plan of Action for ML 
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ANo more “Ratings for Life” 
| Ratings are now valid for only 3 years (April 2007- April 
2010) 


1 SSC-C will lose its CMMI® ML3 rating on 27 April 2010 if 
another Command-level SCAMPIS™ Class Moappraisal is 
not successfully completed before then 


ASustain the current Command-sponsored projects 
(representative sample) 


ASelf-Assessments/Appraisals i mentoring and coaching of more 
projects 
APlan for and Implement 
1 CMMI® V1.2 (CMMI®1 DEV) New Model 
1 Maturity Levels 4/5 
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ATake a fresh look at the entire measurement program with 
an eye towards managing the projects using quantitative 
data 


ACollect and evaluate project historical data for measuring 
cost, schedule, and quality 


AEstablish a process for maintaining the appropriate data 
to begin managing quantitatively 


1 Select at least one main contributorOsub process per project 
lifecycle phase, at least one project management sub process 
and at least one support sub process 


AStatistically manage the data 


1 Using statistical methods (e.g., Statistical Process Control charts, 
histograms, trend charts, etc.) 
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ADemonstrate stable historical data for measuring cost, 
schedule, and quality 


1 Stable data will help you answer questions like: 
ACan you predict where your next data point will fall? 


ADo you know what your baseline is for cost/schedule 
performance? 


Als your product quality what you expect it to be? 


AAre you finding fenoughddefects before the customer gets 
the product? 


1 As a guideline, strive for at least 4 consecutive data points 
within your established control limits 
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AFormalize performance baselines for the project and 
provide baseline data to organization 


ARe-establish quantitative objectives (for example) 
| Reduce cost variance to +/- 5% 
1 Reduce schedule variance to +/- 10% 
| Reduce delivered defects by +/- 10% 
1 Improve major saves found in peer reviews by 20% 


AUse baselines and variance to predict future 
performance 


AKeep up the ML2 and ML3 process performance! 
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AMay — Dec 2 

| Developed Process Improvement Plan for ML4/5 

| Developed Detailed Schedule for ML4/5 

| Developed QPM Plan Template 

| Held various ML4 Meetings with projects 

| Held SCAMPIS™ for one project using CMMI® v1.2 
ASeptember: Project achieved ML3 

| Increase usage of tools across departments/projects 

| Add additional plans to ePlan Builder as needed 

| Continue internal CMMI® Level 3 mini assessments 


Begin Maturity Level implementation 
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AMay-Dec2 con't 


| Enhance/Expand OMR 


AMore Quality Data from Peer Reviews, Testing Phase and 
Defects from Production 


AMore Statistical Process Control (SPC) Charts 


| Command and Department Project Reviews process 
ALook at quality of plans and implementation of best practices 


AReviews of project status by management driven by project 
metrics 


AMore Peer Reviews to measure fsavesO 


| Better tailoring guidance for smaller projects 


Begin Maturity Level implementation 
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t Conduct ML3 SCAMPIS™ Class MAdappraisals for new projects 
1 Conduct ML4/5 SCAMPIS™ Class Moappraisal for one program 


h2 


t Conduct ML3 SCAMPIS™ Class MAdappraisals on other Command 
projects 


Conduct ML4/5 SCAMPIS™ Class Moappraisals on other 
Command projects 


Conduct SSC-C Command-level ML4 SCAMPIS™ Class AAO 
appraisal in April 2010 


= SEIPartner 


WecHsort 


44 Approved for public release; distribution is unlimited (18 OCT 2007) =i (aisi‘ié!O!OOO Technical Software Services, Inc. 


Your complimentary \__ 


use period has ended. 


~ <PDF : | 
= Complete “a Summary 


sae) Features 


Systems Center 
Charleston 


ADecided on Approach i Use CMMI® for Process 
Improvement and Measuring Progress 


AUsing extensive research, determined the Critical 
Success Factorsofor Implementing CMMI® 


ABuilt Plan of Action/Detailed Schedule for Appraisals 


AProvided Training | Systems Engineering, Processes, & 
CMMI® 


AAdvertised Early Successes 


Almplemented Plan Successfully for Phase 17 CMMI® 
Maturity Level 2 and Phase 2 1 CMMI® Maturity Level 3 


1 On schedule, on budget 


ALaying groundwork for higher maturity 
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Any uestions 


Contact Information 


Michael T. Kutch, Jr. Sandee Guidry 

SPAWAR Systems Center Charleston TEC SOFT, Inc. 

Email michael.kutch _navy.mil Email sdguidry techsoft.com 
Phone 2 - Phone - - 
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4" What is a heterogeneous appraisal 


% = What does “organization” mean - defining _ refining 
the term 


f =" Scoping the “organization” 
= Scoping the appraisal 


= Summary - Who gets to hang the “certificate” on the 
wall, who benefits, what are the risks 


CMMI@® is registered in the U.S. Patent and Trademark Office. 
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omogeneous Project eterogeneous Project 


ye / Customer Customer 


Project: | Project 


! Supplier 


Characteristics: Characteristics: 

ASingle supplier (prime contractor) AMultiple suppliers (contractor + partners) 

AMultiple subcontractors - managed by prime AMultiple subcontractors - managed by 
multiple contractors 


How do we ZD- 
appraise this ? —— 
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We know how 
to appraise this 
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Typical Approach 


Customer Everything we need to 
satisfy the CMMI process 
areas is covered 


--+Ky 


. BUT... 


Are you giving the customer 
full value? 


Supplier A 


Sub 


Are you ensuring project 
success? 

Are you projecting true 
project integration? 


Just appraise this part 
of the project 


iy 
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Ignore this part 
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= “Today, a single company usually does not develop all 

7 the components that compose a product or service. 

a More commonly, some components are built in-house 
and some are acquired then all the components are 
integrated into the final product or service. 
Organizations must be able to manage and control 
this complex development and maintenance process.” 


“The problems these organizations address today 
involve enterprise-wide solutions that require an 
integrated approach. Effective management of 
organizational assets Is critical to business success. 
In essence, these organizations are product and 
service developers that need a way to manage an 
integrated approach to their development activities as 
part of achieving their business objectives.” 


te 


From CMU/SEI-2006-TR-008, CMMI® for Development, Version 1.2, p. 3 
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4 * Organization - An administrative structure in which 
«= *\ people collectively manage one or more projects as a 
whole, and whose projects share a senior manager 
and operate under the same policies. 


The “persistent” part of the business 


= Project - A managed set of interrelated resources 
which delivers one or more products to a customer or 
end user. 


The “transitory” part of the business 


“Supplier” “Organization” “Project(s)” 
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A “virtual” organization 


f = Determine what is 


“In-scope” and what is “Out-of-scope” for appraisal 


= Treat each process area as either an “organizational” 
process area or a “project” process area - give the 
same consideration to the generic practices 


= Get buy-in from your organization and the “partner” 
organization (the assumption is that your partner 
organization is at least CMMI Level ) 


= Get buy-in from your appraiser 
= Write it down 
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Appraise as a“ irtual Organization” 
_4@ Appraise the 


. prals Do not 
=> PrimeG ce appraise the 
forganizationd Customer Partner& 
(in-scope) forganizationo 
(out-of-scope) 
Organization A 
Project A -----}----------- Project B 
Sub <1! 
Artifacts Artifacts 
! 3 Include this 
eg MB Pn Pee rProjecto 
A tvirtualoorganization (in-scope) 
J NORTHROP GRUMMAN 
, Cw 


ISER-MLB-PR-07-153 


Your complimentary | 
use period has ended. 


¥ Co mM 0 | ete Thank you for using 


PDF Complete. 


Seq Features Ss Oo T iL 


4" lf process activities primarily benefit the prime 

} organization, then the prime organization is in-scope 
and the partner organization is out-of-scope. E.g. OPF 
process needs and improvements benefit the prime’s 
processes, not the partners 


" If the process area directly serves the project, then it 
is in-scope, regardless of which organization performs 
it. E.g., PP - project planning is for the benefit of the 
project, regardless of which organization does it 


IN SCOPE OUT OF SCOPE 
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4 *= The Engineering process areas are executed on the 
a project 


= Partner follows the Prime’s processes - if not, then the 
Partner can be treated as a subcontractor 


= Evaluate the processes as executed on the Project, 
the produced artifacts, and interview both Prime and 
Partner staff 


Requirements Development 
Requirements Management 
Technical Solution 
erification 
alidation 
Product Integration 
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a" Process Management process areas are executed in 
«= *) the “home room” part of the organization 


= Process improvement activities primarily benefit the 
Prime, so evaluate the Prime and not the Partner 


= Organizational Training Organizational Innovation 
and Deployment address the Prime’s strategic needs 
and business objectives, so evaluate the Prime only 


Organizational Process Focus 
Organizational Process Definition IPPD 
Organizational Training 
Organizational Process Performance 
Organizational Innovation and Deployment 
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4" Causal Analysis Resolution is tied to OPP and OID, 
«= *) soit may be primarily the Prime’s responsibility and 
not in-scope for the Partner 


= Configuration Management, Decision Analysis, and 
Measurement when used as part of Project activities 
are in-scope for both organizations, otherwise they 
are just in-scope for the Prime 


="PP Amay be handled primarily by the Prime or by 
both, and scoped accordingly 


Causal Analysis and Resolution 
Configuration Management 
Decision Analysis and Resolution 
Measurement and Analysis 
Process and Product’ uality Assurance 
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4 * Integrated Project Management has a strong 
rr organizational component, so it may be primarily the 
Prime’s responsibility 


= Project Planning, Project Monitoring and Control, 
uantitative Project Management, Risk Management, 
Supplier Agreement Management are executed in the 
Project environment, so are in-scope for both 
organizations 


Project Planning 
Project Monitoring and Control 
Risk Management 
Supplier Agreement Management 
Integrated Project Management IPPD 
uantitative Project Management 
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= These GPs are “organizational” in nature and are the 


4 responsibility of the Prime 


GP 2.1 Establish an Organizational Policy 
GP 3.1 Establish a Defined Process 
GP 3.2 Collect Improvement Information 


= These GPs are the responsibility of both the Prime 
and Partner for “project process areas” and for just 
the Prime for “organizational process areas” 


—_ 
IS 
® 


GP 2.2 Plan the Process 

GP 2.3 Provide Resources 

GP 2.4 Assign Responsibility 

GP 2.5 Train People 

GP 2.6 Manage Configurations 

GP 2.7 Identify and Involve Relevant Stakeholders 
GP 2.8 Monitor and Control the Process 

GP 2.9 Objectively Evaluate Adherence 


GP 2.10 Review Status with Higher Level Management 
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» 4" The heterogeneous organization Is a “virtual 
«*) organization” made up of components of different 
companies which do not follow the traditional 
supplier-subcontractor relationship 


= Such an organization can be appraised by careful 
scoping of the appraisal - and with concurrence of the 
organizations involved and the appraiser 


= It is the Prime organization who gets the CMMI rating 


= The customer benefits by having a more complete 
appraisal of the project 


= There is a risk that another company can cause you to 
fail your appraisal - which in itself may be a valid 
indication of your ability to manage a project 
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Raytheon 
Intelligence and Information Systems (IIS) 
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Raymond L. Kile, SEI Authorized Lead Appraiser 
Kathryn Kirby, Raytheon IIS Process Assessments IPT Lead 
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Ray Kile has thirty-seven years experience | 


and is an SEI Certified igh-Maturity Lead 
Appraiser and Instructor with a PMI PMP 
certification. e developed the RE IC model 
used widely by the US DOD and contractors 
and is Chief Engineer at The Center for 
Systems Management where he teaches 
Process Improvement, Software Cost 
Estimating, Project Management, and Systems 
Engineering courses. e has participated in 
well over a hundred CMMI SCAMPI A 
appraisals up through Maturity Level and 
ARC Class C-type appraisals. Ray holds a 
BSEE from the United States Air Force 
Academy and a MSEE from the University of 
Missouri. 


Raymond L. Kile 
Chief Engineer 
Center for Systems Management 
Kidwell Drive, Suite 
ienna, A22 2 


rkile csm.com 
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Kathy has twenty-two years experience as a 
multi-discipline engineer and project manager, 
Raytheon Six Sigma™ Expert, Assessment 
Lead for Raytheon Intelligence and Information 
Systems (IIS) Enterprise Process Team (EPT), 
and member of the Raytheon Technology and 
Process Councils. She is a Technology Award 
winner, an Operational Excellence Award 
winner, and a Space Systems Process 
Improvement Award winner. Kathy holds 
degrees in Physics, Astronomy, Mathematics, 
and Music. 


Kathryn. Kirby 
Senior Principal Multi-disciplined Engineer 
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@ Picking a representative sample for an Enterprise appraisal leading to a 
maturity level rating is a multi-faceted problem. ou must pick a sample 
that 

1 Reflects the diversity in the organization 

1 Satisfies the needs of the community-at-large to ensure credibility 

1 Complies with the requirements of the SCAMPI Methodology Description 
Document (MDD) 

| Meets the organizations requirements from both an economic and 
schedule perspective. 


@ Recently the MDD has undergone revision to version .2 and section 
which addresses sampling requirements. 


@ The new MDD requires identifying “critical factors” that 
i use the organizationG standard processes 
1 provides adequate coverage of all the critical factors for the appraisal. 


e At last year’s SEPG a Design of Experiments approach was described 
however, it relied on a statistically valid set of appraisal data as input and 
is not feasible for many organizations. 
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___fan enterprise 


@ Objectives 
1 Determine a set of candidate programs for an appraisal that truly represent 
the enterprise 
1 Determine a manageable representation that allows for an assessment to be 
completed effectively and efficiently 


@ There are fundamental concepts that will allow you to effectively and confidently 
characterize an enterprise. 


| Enterprise Factor Analysis: Determine the characteristics that most accurately 
represent your enterprise using techniques such as Histograms, Pareto Diagrams, 
Weighted Decision Matrix (WDM), statistical computations, Graphical 
representations,é 


1 Enterprise Interactions ID: Identify any factors where a change in 1 factor makes a 
corresponding change in another factor. This may be useful to reduce the total 
number of factors needed to characterize your enterprise. 


| Enterprise Key Factors Selection: Identify the factors which best represent your 
enterprise based on the factor analysis. 


1 Project Selection: Identify programs that best match these characteristics. 


@ This paper will walk through the methodology used by IIS and the analysis 
performed to characterize the enterprise and select the representative sample. 
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CMMiv . ML SESWIPPDSS 2 2 , ADS v .2 
1 Team of ,including SEI igh Maturity Lead Appraiser 
SEI Lead Appraisers 


1 weeks on-site for Class A, interviewees 
llS has over programs in five major product lines, 
with , employees (additional in IT services 


unit not included in this appraisal). 


Work is performed at six primary sites, and multiple 
secondary and customer sites around the world. 


Programs and product lines span multiple work sites, 
but not all programs or product lines are represented 
at all work sites. 


There is a mix of classified and unclassified programs 
at each site and within product lines. 


Programs range from a few engineers maintaining 
small systems or prototyping new capabilities, to 
hundreds of engineers working in a multi-site, 
integrated development environment to create and 
deliver complex mission systems. 


IIS programs are not production systems, such as 
aircraft, appliances, or missiles, so program 
characteristics of the organization or a product line 
can change significantly in relatively short periods of 
time to meet customers’ changing needs. 
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@ The ability to perform a DOE of such an aggregation is an increasingly 
complex problem. 

1 Collecting even a limited set of characteristics, such as 5 or 6, on 300+ 
programs Is a far more time consuming and costly venture then collecting 
a dozen characteristics on 10 programs. 

1 No experimentation will ever occur which changes the precepts of a DOE. 

i Factors can only be used to find a relosestomatch to existing programs. 

1 In dynamic organizations, by the time data is collected for initial 
aggregation, and statistically analyzed for selection of DOE factors, the 
basis for the initial characterization may have changed enough to 
invalidate that basis. 


@ The end result 
1 DOE quickly becomes unfeasible in large, dynamic enterprises even if the 
enterprise can support the labor cost, potential impacts to programs, and 
months needed for the initial collection and aggregation of characterization 
information, and has the knowledge on-hand to perform a large multi-factor 
DOE. 


@ There are, however, fundamental concepts that can be used to find a 
sample of programs that truly represent your enterprise. 
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@ The ability to perform “ sampling” of such an aggregation is 
unwieldy and complex. 


1 Collecting Objective Evidence (OE) on even 1 or 2 process areas on over 
300+ programs is a time consuming and costly venture. 


1 In dynamic organizations, by the time OE is collected, new programs will 
have started which must be added to the collection, and other programs will 
have ended, so the OE no longer accurately reflects the organization. A 
rsampling as of <date>ohelps, but will require substantiation. 


1 Assuming each program provides minimal artifacts for only 1 PA, IIS would 
have needed to provide and manage over 20,426 artifacts. 


1 Culturally, this can also drive the wrong behavior: perfection in 1 process 
area with inadequate or ignored processes in other areas. 


@ The end result 
t 100% samplingoquickly becomes unfeasible in large, dynamic enterprises 
even if the enterprise can support the labor cost and potential impacts to 
programs, and institutes a process to manage the insertion and 
obsolescence of OE and programs in the organization. 
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Identified the characteristics of programs in the enterprise using 
1! Org charts 
1! Monthly Operations Reviews (MOR) Data 
1 2006 Annual Operating Plan (AOP) (financial tracking) 
1 Sit-down session with leaders across the company 


This information formed the basis for characterizing the enterprise, and provided the 
baseline to ensure we did achieve a truly representative sample. 
1 Inability to represent your enterprise may drive you to a rquantityOsolution. 


Adjudicated programs down to a set of programs that were viable 
programs for assessment. Examples of programs eliminated 

1 program ends prior to 9/2006 

1 program award after 9/2006 

| financial challenges 

| capital sales items... 


Analyzed the program characteristics to determine the most appropriate Enterprise 
Key Factors. 


Compared the programs versus Enterprise Key Factors to down-select to 2 
programs to be further analyzed for consideration. 
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@ Implemented a weighted decision matrix to score the 2. programs against the 
factors. 


@ Analyzed combinations of these 2 programs that provided coverage against the 
product lines and primary sites. 


@ Identified risks and potential options. 


@® Reviewed the programs with the Product Line Managers to resolve any concerns 
or issues. 


@ Identified programs for review with the lead appraiser. 


@ Summarized and Reviewed with the appraisers during multi-day meetings 
1 Representative sample selection process used 

IIS enterprise characteristics 

Characteristics of the 14 candidate programs 

Various combinations of the candidate programs against the enterprise 
characteristics 


@ Selected programs as the enterprise representative sample for the appraisal. 
= 22 SS 
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For the initial data collection, a team of experienced appraisers and 
program leaders across IIS identified characteristics used in past 
appraisals, and key characteristics used in day-to-day operations. 
@ Characteristics included 

| Pre-acquisition or Post-Award 
1 Single or Multi-site programs 
Development vs O&M 


CMMI Requirement or existing Objective Evidence baseline. 


| 
| Classified or Unclassified 


® Point of Contact (POC) at each Raytheon IIS site collected this 
information on existing programs using 


1! Organization and Staffing Charts 

1 Monthly Operations Review Data 

1 Sit-down sessions with product area and engineering managers. 
: 


2007 Annual Operating Plan (AOP) (contains the financial 
expectations for all programs in IIS). 
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@ Some categories were too broad and further detail was needed. 
1 Example: Development vs O&M needed to be expanded to help us 
understand the scope of a program: 
1 System Development but no Development of Software Deliverables 
System and Software Development 
Delivered products in O&M with additional development in work 
O&M not a follow-on to development 
Service contract 
Fixed staff level contract 
Time & Materials contract 
Other 


e Other factors were identified that impacted the characterization of the 
enterprise. Examples 
1 Offsite programs that use prime or customer processes. 
1 Some programs are split on-site/off-site, and needed to know relative % of 
on/off site effort. 


@ The data on these factors was then analyzed to select the 2 Candidate 
Programs. 
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@ Multi-site (which sites) 
@ Program office location 
e on-site off-site effort 


@ Program Type 
1 Development 
1 O&M with/follow-on to 
development 
1 O&M nota follow-on to 
development 
| Service contract 
1! Time and Materials 
| Fixed staff 
1 Other 
@ Appraisal Reuse basis 
1 Has an existing objective evidence 
baseline 
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Program Scope SE, SW, 
Subcontracts 


Contractual CMMI requirement 


Product Lines Covered SS, NS, SIS, 
TIS, OTS 


Pre-acquisition proposal program or 
Post-acquisition program 


Classification level and special 
restrictions 


Internal Gate Status (lifecycle 
indicator) 


Current engineering technical staff 
Projected engineering technical staff 
in 2 


AOP Contract value in 2 
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Gate Reached in Sept 2 


In Development or Development 
Proposal and Pre-Bid 


25.84% Post Delivery and Program Closure 


4.49% 
29.21% ; 


The dual peaks and number of 
programs between gates 5 and 
9 reflects spiral development 
methods. 


Observations 


Gate 3 Gate 4 Gate 5 Gate 6 Gate 8 Gate 9 Gate 10 Gate 11 


Gate 5: Program Startup Review Gate 9: Test/Ship Readiness 


The 2-hump histogram is an artifact of predominant spiral developments. 
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Production Study 


Service 


2% 


3% Development 
O&M with No 47% 
Development 
3% 
Task order 
3% 
Other 
3% fF 
Lal 
Materials 


2.68% 


0 
= 3.36% 


2.01% 
2.01% 
1.34% 


O&M with 71.14% 28.86% 
Development 
20% 


O&M Non-Dev 0.00% 


Fixed Staff 
12% 
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Relationship Analysis AOP vs_ Technical Staff 
Focused Moving Average 


Correlation Coefficient (Full data set) 


Relative Contract alue 


Technical Staff 


Some correlation exists between technical staff and AOP, but not enough 
to allow us to exclude either the AOP or Technical Staff factor. 
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Key factors that were used to select the 2 Candidate Programs. 


@ Multi-site (which sites) 
@® Program office location 
on-site off-site effort 


@ Program Type 
1 Development 
1 O&M with/follow-on to 
development 
1 O&M nota follow-on to 
development 
1 Service contract 
1 Time and Materials 
1 Fixed staff 
1 Other 
@ Appraisal Reuse basis 
1 Has an existing objective evidence 
baseline 


Program Scope SE, SW, 
Subcontracts 


Contractual CMMI requirement 


Product Lines Covered SS, NS, SIS, 
TIS, OTS 


Pre-acquisition proposal program or 
Post-acquisition program 


Classification level and special 
restrictions 


Internal Gate Status (lifecycle 
indicator) 


Current engineering technical staff 
Projected engineering technical staff 
in 2 


AOP Contract value in 2 
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Multi-site programs demonstrate we operate as an enterprise, not individual sites. 
Primary sites coverage demonstrates a majority of our work locations. 
Coverage of each Product Line ensures we include all our major customer groups. 


On-site programs present fewer logistics issues and ensure minimal impact to our 
customers. 


Larger development programs demonstrate our processes over a broader scope of 
the model than a study or short duration Time Materials contract. 


Later lifecycle programs can demonstrate processes over a broader life-cycle scope. 


Programs with engineering development as well as subcontracts demonstrate our 
processes over a broader scope of the model. 


A contractual CMMI Program Requirement helps identify our commitment to our 
customers and our contractual obligations. 


Previous SCAMPI provides an OE refresh baseline to reduce risk against our 
aggressive schedule, and ID areas where more training and collection effort will be 
required. 


O M in addition to on-going development demonstrate our processes over a broader 
scope of the model as well as commitment to maintain solid development processes. 


Larger programs better represent IIS against the 2 AOP. 
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s. Product Line AOP 


Percent of Total 
AOP % of IIS 
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many combinations analyzed) 


In our sample selection review, the appraisal team 
Product Lines Coverage was able to make real-time changes to our 
recommendation, review the permutations against the 
Enterprise characteristics, and identify the programs 
be included in the final sample set. 


Number Of Programs s. Product Line AOP 


30.0% 
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25.0% mAOP% per lls 


OTS SIS 
Product!” 


Programs 


20.0% 
15.0% 
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Primary SitesCoverage 
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0.0% 
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Product Line 
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Church College 
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liS Engineering by Site 
As of October 2 


State 
Reston College 


Falls Linthicum 
hurch 
enue Omaha 2 


Garland 


2 Aurora 


The largest percentage of the engineering population is 
located in Aurora.... 
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s. Product Line AOP e NS program 


2 programs to better 
represent the largest 
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Halle 


Produce Line 


0 Percent of Total e OTS p rog ram 
a AOP % of IIS 


e SIS program 


e@ SS 2 programs (larger 
engineering population) 


TIS program 


PT Enterprise Process 
pization 


Product Line Coverage 
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@ Applying a solid process in your enterprise characterization and selection 
of a representative sample will allow you to 
1 Meet the requirements of the MDD 
1 Execute an enterprise appraisal against a reasonable representative 
sample 
1 Maintain the integrity of the appraisal and confidence in the results. 


@ Choosing wisely will take time. 


@ Using the DAR process area, the GP’s, and your own internal best 
practices for decision making will help guide you though the enterprise 
characterization and selection process. 


@ Collecting and analyzing the program factors of the enterprise, and 
reviewing the results with the appraisal team will allow you to 
1 Provide your appraiser with a deeper and broader understanding of the 
enterprise 
1 Save time during the execution of the appraisal by reducing questions and 
issues related to how the enterprise operates. 


@ Ensuring you have solid depth and breadth in an enterprise 
characterization and selection of your representative sample will make it 
easier to complete the ADS v _ .2 —- you will have the data on-hand. 
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Kathy has 22 years 
experience as a multi- 
discipline engineer and 
project manager, Raytheon 
Six Sigma™ Expert, 
Assessment Lead for 
Raytheon Intelligence and 
Information Systems (IIS) 
Enterprise Process Team 
(EPT), and member of the 
Raytheon Technology and 
Process Councils. She is a 
Technology Award winner, 
an Operational Excellence 
Award winner, and a Space 
Systems Process 


Improvement Award winner. 


Kathy holds degrees in 
Physics, Astronomy, 
Mathematics, and Music. 


Kathryn H. Kirby 
ee IS EPT CMMI Team 
L 


ea 
khkirby@raytheon.com 


John has performed activities 
related to training, appraisals, 
and consulting for many years 
in the areas of program and 
project management, 

systems engineering, and 
process improvement. He has 
degrees in business, physics, 
and engineering. 


John is an SEI Authorized 
High-Maturity SCAMPI Lead 
Appraiser as well as an 
authorized SEI Instructor. 


John B. Miller 

SEI Authorized High-Maturity 
Lead Appraiser 

IIS EPT CMMI Expert 


jomiller@raytheon.com 
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Carolyn is the IIS EPT Process 
Architect, developed the vision 
for an IIS-wide Common 
Process Architecture, and was 
key to the success of the IIS 
Enterprise CMMI 
SE/SW/IPPD/SS ML3 rating in 
2006. She established a 
successful architecture for 
AuroraG pursuit of a CMMI 
(SE/SW/IPPD) Level 3 rating in 
2004; unique in that Aurora was 
the only Raytheon site to earn 
the IPPD distinction, was a Peak 
Performer winner in 2004, and 
has proven that defined 
processes lead to better 
business. Her vision for a 
Common Process Architecture 
across IIS is now a reality. 


Carolyn B Hickey 
IIS EPT Process Architect 
CMMI Appraiser 


cbhickey@raytheon.com 
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Overview of the Raytheon IIS Enterprise 


Key Considerations and Objectives in the Appraisal 
Approach 


Critical Path to the IIS Enterprise Level ML3 
SE/SW/IPPD/SS SCAMPI 


Appraisal and Support Teams 

Representative Sample 

Objectives and Scope of the Appraisals 

Run Rules, Roles & Responsibilities, and Expectations 
Managing and Preparing the Objective Evidence 
Orchestrating and Conducting In-briefs and Interviews 


Managing the appraisal schedule, input, plan, and the new 
SAS 
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e CMMiv. ML SESWIPPDSS 2 2 , ADS 
v .2 
e& Teamof ,including SEI igh Maturity Lead 
Appraiser SEI Lead Appraisers 
E weeks on-site for Class A, interviewees 
e IIS has over programs in five major product 
lines, with , employees (additional in IT 


services unit not included in this appraisal). 


= Work is performed at six primary sites, and 
multiple secondary and customer sites around 
the world. 


= Programs and product lines span multiple work 
sites, but not all programs or product lines are 
represented at all work sites. 


= There is a mix of classified and unclassified 
programs at each site and within product lines. 


= Programs range from a few engineers maintaining 
small systems or prototyping new capabilities, to 
hundreds of engineers working in a multi-site, 
integrated development environment to create 
and deliver complex mission systems. 


e IIS programs are not production systems, such as 
aircraft, appliances, or missiles, so program 
characteristics of the organization or a product 
line can change significantly in relatively short 
poling of time to meet customers’ changing 
needs. 
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® Clearly defined business needs and objectives. 


® Perform appraisal preparation activities concurrent with the 
integration of the IIS Enterprise Common Process. 


® Incorporate Lessons Learned related to enterprise-level 
appraisals. 


= Perform assessments of processes and programs as early 
as feasible. 


® Reduce risk by scheduling multiple preparatory appraisals 
with clear objectives. 


& Identify internal best practices to be used to plan, manage, 
and execute the appraisal effort. 


e Integrate the Raytheon Gate Review Process with the Appraisal 
Readiness Review process to ensure readiness for the next step. 


= Use Raytheon Six Sigma™ process improvement methodology. 


Your complimentary 


use period has ended. 


F Complete ™":y2¢omiee ye IS Enterprise Level ML Raytheon 
re to . C AMPI Customer Success Is Our Mission 


FQuick- 
Lookso 


Level 3 
Org 
Process 
Class cl 


Level 3 Level 3 Level 3 
SE/SW/IP SE/SW/IP SE/SW/IP 
PD/SS PD/SS PD/SS 

Class B2 


Level Continuous 
SE SW IPPD SS Process 
SCAMPI Improvement 


Appraisal Related Focus [_ Common Process Focus [_ 


Multiple Events with Clear Objectives Combined with Raytheon’s Gate Review 
Process Helps Gauge Appraisal Readiness. 
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Program 
Appraiser Site POC Appraisal 
Work Site Lead POC (OE Team Lead) “ am Session Monitors| Mini-Team 
Primary Appraiser Session Monitor 
Work Site 1 | assignedto | Site x POC OE Team Lead Mini-team Lead 
other sites Program 1 other sites Program 1 POC Program 1 session monitors Member #2 
Primary Appraiser Session Monitor 
Work Site 1 | assigned to | Site y POC OE Team Lead Mini-team Lead 
other sites Program 2 other sites Program 2 POC Program 2 session monitors Member #2 
a 


= To maintain objectivity, the Appraiser Lead for a program was not on 
the Mini-team for the Class B or SCAMPI Appraisals. 

& The Appraisal Coordinator was also the Appraisal Lead and the 
Designated Agent for the Lead Appraiser in the new SEI SAS tool: able 
to keep all areas in-step. 

= Defining the skills needed for OE collectors is crucial, but very difficult. 

er We would love to learn from your examples! 


The Right Person Makes a World of Difference. This Applies to All Positions, not Just 
to Your OE Teams. 


The use of Four SEI Authorized Lead Appraisers on the Nine-person Appraisal Team 
was Critical to Executing an Appraisal of this Magnitude to Schedule. 
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® Identified the characteristics of programs in the Enterprise using Org 
charts, program and function review requirements, business plans and 
financial targets, etc. 

® Characterized the IIS enterprise using statistical tools and underlying 
concepts of a Design of Experiments (DOE). 

® Identified 178 programs that were viable programs for assessment. 
Ex. Eliminated programs ending before 9/06, capital sales. 

=m Analyzed the program characteristics to determine the most 
appropriate Enterprise Key Factors. 

= Compared the 178 programs versus Enterprise Key Factors to down- 
select to 20 programs to be further analyzed for consideration. 

® Rated the 20 programs using a QFD and reviewed the results with the 
Product Line Vice Presidents to identify risks or potential issues. 

® Identified 14 program which were then summarized and reviewed with 
the Appraisal Team. 

® The appraisal team selected seven programs as the enterprise 
representative sample for the appraisal. 


A Solid Analysis to Select and Use Enterprise Key Factors is a Viable Method for 
Selecting Representative Programs. 


Methodology Presented at the NDIA CMMI 2 Conference, Denver CO ( 2) 8 
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Class | AEnsure IIS level | AAIl policies and - days - Raytheon-only assessment 
policies and procedures at IIS level per team 
procedures are for all CMMI process function | - No Interviews 
complete areas. - Function will not proceed 

A Ensure IIS AOE for OPF OPD to Gate until all issues 

procedures are are addressed. 
compliant with 
the CMMI. 


Class | A Focused review | AEnterprise processes - Lead Appraiser and full 

of Process and procedures Appraisal team 
Areas AAI| SCAMPI Target - Will proceed like a Class A, 

A Use appraisal Programs but without determining 
procedures to AProgram tailoring, goal satisfaction 
identify program processes and 
necessary program procedures 
improvements ADirect and Indirect 

A Determine Objective Evidence 
readiness for Alnterviews related to B 
Class A SCAMPI Process Areas 


The Objectives and Scope of each Appraisal were Defined to Meet the Specific 
Business Needs and Objectives Defined in our Appraisal Approach. 
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= Define early, use early, use often, repeat yourself constantly, 
and update based on lessons learned: 
e Objective Evidence Standards, Practices, and Management 
®& SCAMPI Program Expectations and Responsibilities 
= SCAMPI Participants6Expectations and Responsibilities 
Executive Management Expectations and Responsibilities 


® Independent Review Team/Gate Review Team Overview, Gate Scope, 
Summary Status 


® Process and Run Rules for Data Requests 
Run Rules for Appraisal Sessions 


> Lesson learned from the B1 appraisal led us to add Session Monitors to all 
activities: in-briefs, out-briefs, interviews, findingsé 


®& Appraisal Team Member Information Package 
Appraisal Support (IS, POCG,é ) and Programs Contact Information 


Items not Defined Formally, Well Enough, or Soon Enough, Cost us Schedule and 
Budget. 
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® Established OE color code status and run-rules based on 
separate phases in our appraisal lifecycle: 
e Initial OE population and review by the appraiser 
® Class C Appraiser Review 
® Update of the OE after C Appraisals 
® Class B Appraisals 
= Update of OE after Class B Appraisals 
® OE review as part of the Class A (in accordance with the SEI MDD) 


& Document, train, and enforce OE process 
® Currently improving our process based on lessons learned 


Just like a Program, as Your Life-cycle Changes, the Way You Status and Manage 
Must Change As Well. 


Trying to Merge OE Mgmt from Multiple Places into One Enterprise“Make Everyone 
Happy” Model Yields a Non-Standard That Meets No Need Well. 
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® Session Monitors: 
® Assigned at each site for each session 
® Initiated the telecon and web conference sessions 
® Confirmed attendance 
e Held attendee electronic devices or other items 
er Remained outside the conference room 
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e Ensured all parties were ready, turned the session over to the 


appraisers, and left the room. 
me The IntervieweeG perspective: 


e In the room: better via telecon; less pressure and more comfortable. 
®& On the phone: better in the room for body language and feedback. 


Sessions Can Be Successfully Conducted using Web/teleconferencing and tools, but 
Ensure You Meet the Strictures of the Model and the Requirements of your Lead 


Appraiser and Appraisal Team. 


Don’t use the same Telecon number for Back-to-Back Sessions, or Use a Telecon 


Number Commonly Used for Other Meetings. 
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® Created, managed, and controlled the Appraisal Input and 
Plan outside of the SEI Appraisal System (SAS). 
® Information had to be highly condensed to fit the SAS text field limits. 
e SAS-external documents simplified mgmt review and sponsor approval. 


® We used three schedules to manage our appraisal events. 

® A program Integrated Master Schedule (IMS) with EVMS. 

e® A general distribution day-to-day schedule showing session titles, 
dates, times, and sites involved. 

e A restricted access detailed schedule that added the participant names, 
appraisers, session monitors, conference rooms, and telecon numbers. 

+ All employees had insight and maintained involvement without violating 
confidentiality. 

- Appraisal Coordinator, Appraisal Team, Site POCs, Program POCs, & Session 
Monitors had to manage to multiple schedules. 


Seek Ways to Manage using One Detailed Schedule with a Restricted Distribution, but 
be able to Quickly Extract a “Clean” Version. 
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Presentation Objective 


saermiaei) features 


A Explore some common perceptions about the SEI CMMI 
and the IEEE Life Cycle Standards 


“* CMMI® can be leveraged for IEEE Standards 
¢* IEEE Standards support CMMI®-based process improvement 
A Define a feasible approach for using the SEI CMMI and the 
IEEE Life Cycle Standards together 
“+ Key questions: 
1 How to exploit the synergy? 
| How to resolve the differences? 


Life Cycle Standards to be covered in this presentation will be referred as 
IEEE IEEE 22 _ , Software Life Cycle Processes 


IEEE IEEE 2, IEEE Recommended Practice for Software Acquisition 
IEEE 2 IEEE 2 _ , System Life Cycle Processes 
IEEE 2 IEEE , IEEE Standard for Developing a Software Project Life Cycle Process 


® CMM is registered in the U.S. Patent and Trademark Office by Carnegie Mellon University 
* EEE - Institute of Electrical and Electronic Engineers 
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A Two key elements of successful acquisition of software- 
intensive systems 
“* Selecting the right suppliers 
| Capability assessment of the potential suppliers is a key 
element of the acquisition process, and the current, widely 
embraced recommendation is the reliance on the CMMI [SEI 
2007] 


«* Assuring mission success 
1 Aerospace experience shows that mission success is achieved 
via the use of robust development standards [Eslinger 2006]. 


| Eslinger demonstrates that even the use of so-called “mature” 
processes, such as the CMMI _ is inadequate, and the 
government must make a robust software standard contractually 
compliant 


“¢ It seems that we need both the CMMI® and the IEEE 
standards 


‘ Note that Eslinger’s development standard recommendation is based on IEEE 12207 
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- Supplier (Contractor) Perspective 
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A It seems that we need both the CMMI _ and the IEEE standards... 
<* CMMI® 
1 It is the de-facto process improvement standard 
“+ IEEE Standards 


1! The government is using them to define a framework for development 
planning and engineering 


A Contractors’ main concerns 
“* Finding the most effective/efficient ways to ensure dual 
compliance 
“* Be pro-active and prepared for rapid tailoring of the standards 


1 Simultaneously with compliance though, need to find out how to 
ensure agility and competitiveness from the development process6 
perspective 


“+ Last but not least, to make a profit 
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A “Weare from the SEI and the IEEE and we are here to 
help you” 


--- Paraphrased use from Ronald Reagan 


A Standards are always out of date. That s why we call 
them standards.“ 


--- George F. Will 
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A Regarding “ elp”, all the mentioned sources have noble 
goals however, their objectives are different ... 
“¢ CMMI® 
1 The objective of CMMI® for development is to help 
contractors improve their development and maintenance 
processes for both products and services 
1 Based on [Chrissis 2007] 
“+ IEEE Standards 
i IEEE Standardséobjective is to eliminate misunderstandings 
between contractors and procurers 
1 Based on [IEEE 1997] 
“* ISO Standards 
i ISO Standardséobjective is to promote a free and fair global 
trading system via worldwide standardization 
| Based on the ISO website [ISO 2007] 


*1SO — International Standards Organization. Note that the discussed key life cycle standards have their origin in ISO standards 
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(RR 
A The typical, current IEEE standard development process has 
some inherent characteristics 
«+ IEEE Standards are developed in isolation 


| Authors in most cases give lip-service only to other IEEE standards 


1 The CMMI® has only minor, vague references to IEEE Life Cycle 
Standards 


1! The IEEE Life Cycle Standards have only some vague references to 
process improvement (nothing specific regarding the CMMI®) 


“+ IEEE Standards are developed by volunteers 
«+ IEEE Standardsétext is finalized and balloted via consensus 
A As aresult, most standards share some common characteristics 
“* The final material is always a result of major negotiations 
«“* The process takes several years 
«+ The standards are not consistent with each other 


“* Standards | at most! codify state-of-the-practice, and never 
reflect the state-of-the-art 


A The CMMI has not been developed by the IEEE, but with respect 
to these problems it is not very different 
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A The use of IEEE standards only makes sense for 
compliance 
“* Sure, they can be used as guidance materials by 
novices, but better, more up-to-date, and widely 
accessible materials are available for instructional 
purposes 
A The CMMI is also about compliance 
“* It is claimed that the model is for guidance, and only 
describes rwhatdtype required characteristics for process 
improvement, without prescribing the fhowa However, 
1 Unconditional satisfaction of all goals is required on every 
maturity level 
1 In reality, during appraisals when the level of 


institutionalization is determined, the level of compliance 
with all the pre-defined organizational processes is probed 


It is really difficult to simultaneously satisfy so many conflicting 
compliance requirements 
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A l\EEE 2 
“+ The use of 1062 is not recommended for government 
acquisitions 
1 Various government entities have their own, strictly binding 
acquisition policies and instructions (e.g., the DOD 5000 
series, NSSAP 03-01, etc.) 


«+ Acquirers are better off with directly using their respective 
policies 
A IEEE life cycle standards have serious technical, 
compliance, and tailoring problems of their own 
A IEEE life cycle standards are poorly harmonized with 
each other 
“+ There are some harmonization efforts in progress, but no 
tangible results yet 
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seaeriaei) features 


Organizational terms and their hierarchy 
IEEE 2 IEEE 22 IEEE CMMI 


| Enterprise | Organization | Organization | Organization 


A Concerns 
“* The organizational terms and their hierarchical relationship seem to 
be the same for the IEEE Standards and the CMMI® (IEEE 152886 
rEnterprisedis the sole exception); however, the underlying 
definitions are different 
| E.g., the CMMI® definition of an rorganizationdis the most constrained 
«* All standards allow for recursive invocation of these terms 
t é& but they doné provide guidance on the details 
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»  _____ fganizations and Projects 
DE re 
A “Project” across the IEEE Standards is not well defined 
¢* In 1074 the term is used to limit the standard@ scope 
1 It only implies an endeavor in conjunction with software 
development and maintenance 
A “Organization” in the IEEE Standards is also only 
vaguely defined 
“+ In general, it refers to the environment (people and 
facilities) where the fprojectsoare executed 
A “Project” in the CMMI refers to a managed set of 
interrelated resources delivering one or more products 
to a Customer 
A “Organization” in the CMMI refers to an administrative 
structure where projects share a Senior Manager and 
operate under the same policies 


* “Customer”: The party accepting the product or authorizing payment 
** “Senior Managers” focus on the long-term vitality of the organization rather than the short-tem pressures of the projects. 
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A The first step is always to develop the WBS 
“+ There must be at least one product 
A Next, map the WBS to contractor relationships 
¢* Where there is a Contract, there is a Customer and a Supplieré 
“* This will also help to determine what should be considered as Projects 


“* For determining Organizations, use the existence or lack of Senior 
Managers (by CMMI® definition) to draw up the organizational boundaries 
for process development, maintenance, and improvement 


A The WBS can also be used to determine which standard to use and where 
“* 15288 for Systems Engineering projects 
“* 12207 and 1074 for Software Engineering projects 
A Discipline coverage 
“> The CMMI® covers Hardware, Software, and Systems Engineering 
“+ IEEE process standards cover only Systems Engineering and Software 


“+ Note that there are no applicable, true hardware process standards 
(Hardware development processes are not Life Cycle Model-driven like 
Systems or Software Engineering processes) 


Clear definition of Organizations is critical because both the IEEE Standards 


and the CMMI are referring to so-called Organizational Process Assets 
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A The process to develop a SW Project Life Cycle Process in 
IEEE and the process to develop a Defined Software 
Process inthe CMMI look similar , but there are subtle, 
critical differences 

“* IEEE 1074 only specifies Activity Description, Input Information, 
and Output Information for every Activity 

“* However, a defined process in the CMMI® must clearly state 
the following: 

1 Purpose 

1 Inputs 

1 Entry criteria 

1 Activities 

1 Roles 

| Measures 

1 Verification steps 

1 Outputs 

1 Exit criteria 


* Note backup slides showing the details of both processes 
** See [Chrissis 2007], page 152-154 for the distinctions between a managed and a defined process in the CMM/® 
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(Re 
A Interms of reliance on Organizational Standard Processes the 
CMMI is stricter than the IEEE Standards 
“+ IEEE 1074 does not mandate the use of any existing SPLCP 
(Software Project Life Cycle Process) during the creation of the 
current SPLCP (It doesné assume their existence) 


1 In fact, the recommended Organizational Process Assets are more 
Supporting than defining elements of the rio be createdOSPLCP 
(e.g., policies, metrics, tools, methodologies, etc.) 
«* A CMMI® Defined Process must be tailored from the OSSP 
(Organizational Set of Standard Processes) 


* Note backup slides showing how IEEE 1074 and the CMMI? are dealing with Organizational Standard Processes 
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A It makes a difference on what CMMI maturity level the 
organization is when the decision is made to comply with the 
IEEE Standards 

«* For a Level-2 organization it is not a problem 
1 SPLCP created with the use of the IEEE Standards immediately 
satisfies the conditions for a Managed Process 
1 Eventually these kind of processes could become part of OSSP 
** For a Level-3 or higher maturity organization there is a conflict 


1 Newly created, IEEE Standards-based processes can not be 
immediately applied because first they would have to be made 
part of OSSP 

| Also, appropriate tailoring guidelines would have to be 
developed and documented 

1 The new processes can also affect (override) existing sub- 
process selection for quantitative management 


1 The undesired side-effect is administrative delays and overhead 
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A Good News 
“¢ There is no difference in how the IEEE Standards and the CMMI® 
treat Life Cycle Models 
A Bad News 
“* The same © 
A Both expect the availability of a collection of life cycle models 
“* CMMI® 
1 OPD (Organizational Process Definition) SP (Specific Practice) 1.2: 
Establish Life Cycle Model Descriptions 
“* 1074 
1 Clause 4.2.1 shows the existence of a collection of SPLCMG, but it 
also declares that this collection is out of scope for the standard 
A Nevertheless, neither the CMMI nor 
| Specifies where these models supposed to come from 
1 Specifies how they should be documented 
1 Provides guidance on tailoring 
A IEEE Standards do provide some guidance on selecting a life 
cycle model for a project, e.g., 22 for software 
“se ee guidance is for a limited number of Life Cycle Models 
only 
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A Mappings have been carried out on the CMMI Process Area level 
% CMMI®Y 15288 (Coverage = 68%) 
% CMMI®Y 12207 (Coverage = 72%) 
% CMMI®&Y 1074 (Coverage = 59%) 
A \EEE Standard support leverage by CMMI Process Area Category 
“«* Basic Process Management 
| Weak 
** Advanced Process Management 
1 Missing due to lack of quantitative management focus 
* Basic Project Management 
| Well covered 
“+ Advanced Project Management 
| IEEE Standards do not provide robust enough support 
“* Engineering 
| Well covered 
“+ Basic Support 
| Adequate coverage, except for Measurement Analysis 
“+ Advanced Support 
| Partial coverage due to lack of quantitative management focus 


* See detailed mapping at the end of the presentation in the Backup Slides section. 
Definition of Coverage [%] = (22 - > “Not covered” / 22) * 100, where 22 is the total number of CMMI® Process Areas 
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A Should the IEEE Standards be evaluated (mapped) ona 
lower, CMMI practice sub-practice, level 
** No, practices and sub-practices are not required model 
components 
1 Practices are only expected 
1 Sub-practices are only informative 
“+ Process tailoring must be based on specific business 
goals and mission objectives 


1 Equivalency/Adequacy of actual process steps must be 
assessed on a case-by-case basis 

1 Early attempts for low-level evaluations in absence of 
specific business goals and objectives could be educational, 
but do not provide an effective/efficient solution 
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plementation Pathways 
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A (_) Introducing IEEE Standards in a high maturity environment 


«* The introduction of IEEE Standards-based processes is well facilitated by 
the robust process development, maintenance, and improvement 
infrastructure 


«¢ As it was discussed, there is some ambiguity related to the CMMI® 
interpretation of a Defined Process, but mission success and the use of 
robust development standards should be the primary concern 


A (2) Introducing CMMI where IEEE Standards are the norm 
«+ IEEE Standards-based processes provide a good starting point 


“* Pay attention though to the much stricter process documentation 
requirements of the CMMI® 


A ( ) Ideal situation would be simultaneous introduction 
«* Use business and mission objectives as primary tailoring drivers 
“¢ Pay attention to the CMMI® Context 
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A \EEE Standards support CMMI -based process improvement 
«* IEEE Standards can be helpful in creating both managed and 
defined CMMI® Process Assets 
“+ owever, be prepared that their practical application is 
time-consuming and requires effective tailoring 
«+ Another caveat: IEEE Standards are weak and need special 
attention in the following four areas when introduced to support 
the implementation of CMMI®-based process improvement: 
1 Definition of the process management infrastructure 
1 Measurement & Analysis Process Area 


1 Characterization of defined processes, due to 
1 The need for more detailed documentation that provides greater 
insight into process activity relationships 
| More rigorous execution requirements 
| Process improvement focus 
1 Quantitative Management 


A CMMI can be leveraged for IEEE Standards 
“* Tailoring of IEEE Standards is more efficient if it is done with the 
understanding of CMMI® requirements and terminology 
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Capability Maturity Model Integration 
Carnegie Mellon University 
Commercial Off-The-Shelf 
Center for Software and Systems Engineering (at USC) 
Department of Defense 
ardware 
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National Security Space Acquisition Policy 
Organizational Process Assets 

Organizational Process Definition (CMMI Process Area) 
Organizational Set of Standard Processes 
Project’s Defined Software Process 

Planning, Programming, Budgeting, and Execution 
Software Engineering Institute 

Specific Practice (of a CMMI Process Area) 

SW Project’s Life Cycle 

SW Project’s Life Cycle Model 

SW Project’s Life Cycle Process 

Standard 


PPBE 


STD 


United States Air Force 
University of Southern California 
Work Breakdown Structure 


__NSSAP | National Security Space Acquisition Policy 
____OPA| Organizational Process Assets 
____OPD | Organizational Process Definition (CMMI_ Process Area) _ 
___ossP| Organizational Set of Standard Processes 
___PDSP | Project’s Defined Software Process 9 
___PPBE | Planning, Programming, Budgeting, and Execution 
____SE!| Software Engineering Institute 
____SP| Specific Practice (ofa CMMI Process Area) 
___SPLC|SW Project’s LifeCycle 
__SPLCM | SW Project’s LifeCycleModel 
__SPLCP | SW Project’s LifeCycle Process 
_STD|Standard 
__SW| Software 
___USAF | United States AirForce 
____USC | University of Southern California 
____WBS | Work Breakdown Structure 
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A Enterprise Processes A Technical Processes 
«* Enterprise Environment “* Stakeholder Requirements 
Management Definition 
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* Requirements Analysis 
“* System Life Cycle Process 
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A 22 Software Life Cycle Processes A Software Activity Groups 
“+ Primary Life Cycle Processes “+ Project Management 
1 Acquisition 1 Project Initiation 
i Supply 1 Project Planning 
1 Development 1 Project Monitoring and Control 
i Operation «+ Pre-Development 
1 Maintenance 1 Concept Exploration 
“+ Supporting Life Cycle Processes i System Allocation 
1 Documentation 1 Software Importation 
Configuration Management «+ Development 
Quality Assurance 1 Software Requirements 
Verification 1 Design 
Validation 1 Implementation 
Audit Installation 


Problem Resolution 
“+ Organizational Life Cycle Processes 


Operation and Support 
Maintenance 


I 

‘ 

‘ 

‘ 

‘ 

i Joint Review «+ Post-Development 

| ‘ 

| ‘ 
‘ 

| ‘ 


Management Retirement 
i Infrastructure «+ Support 
1 Improvement i Evaluation 
Training 1 Configuration Management 
1 Documentation THE AEROSPACE 
CMMI 2 -— Peter antos Slide 2 “ 
ide a CORPORATION 


Training 


Your complimentary a 
use period has ended. 


E,complete cima 


Project Life Cycle Process in 


spasm) features 


Activity Group Activity Group, ActivityGroup Activity Group Activity Group 


Information Flow 
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System WBS (Work Breakdown Structure) 


freee features 


Conceptual WBS Space System Terms 
. 


Software-Intensive System 


Mixed Subsystems 


sbsysers J 


W Components SW Components 


LEGEND 
Hi Software 


[]  ardware 


“Activities” in WBS jargon refer to the creation, 


integration, and management of the product 
elements on every level 


ff] Activities 
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Example (Simplified) Space System Contractor Relationships 


Contract 
including IMP (Integrated Master Plan) 


Commercial 
Customers 


Contracts 
Prime Contractor 


Government | 
Sector 


Commercial Program Ground 
zation [aiilicontrector Ea > —===—"> Segment 
Organization Contractor Office Contractor 
Internal liternal Development 


Formal Formal 
Agreement a Agreement 


External Su bcontract 
Subcontractor ssccsseusccsecsecrgereee 


Program Office Subcontractor 
and Space Segment =~ ° “contractor Ground Segment 


Payload Development Internal Development 
| Formal 
Agreement 


The product’s WBS and the contractor relationships together will 
determine the actual organizational structure (See next slide) 
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(Simplified) Space System IPT (Integrated Product Team) Structure 


Program Level 
IPT 


Systems 
Engineering and 
Integration Team 


IPT 


Ground Element | Ground 
IPTs Integration Support 


ardware | 
IPTs 


Color Legend 
[| Prime Government Sector 


Ground Segment Space Segment 


COTS only 


(Ba Prime Commercial Sector 


[| Sub Division A 
[| Sub Division B 


The challenge What are the “organizations” and what are the “projects” 
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Process (SPLCP) 
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Description Tailoring Library of Process 
of LCM’s Guidelines Documentation 
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Organizational Set 


of Standard 
Allocate Processes (OSSP) 


System 
System ——> nae eeellEeelllllUrhCl Allocated SW 
Requirements Requirements *\ Requirements 
to Software 


Select Software 
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CMMI Process Areas Related 2 Life Cycle Processes 
|BasicProcessManagement | 


Po Organizational Training | al 
[Advanced ProcessManagement | 
| Organizational Innovation & Deployment | I | 
[Basic ProjectManagement | 
Project Control 
[Advanced ProjectManagement | 
Integrated Project Management 
| Quantitative Project Management | No d =| 
[Engineering 
| CC RequirementsManagement] 
O 


|[BasicSupport. 
| Measurement and Analysis | N d Z| 
[Advanced Support fF 


Causal Analysis and Resolution | N 


Validation | Validation 
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CMMI Process Areas Related 22 Life Cycle Processes 
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Causal Analysis and Resolution | I 
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CMMI Process Areas Related Activity Groups 


Basic Process Management aga aa 


Organizational Process Definition | Not covered 
Organizational Process Focus | Not covered 
| 


Organizational Training | Not covere: | 
[AdvancedProcessManagement | 
Not covered 
| “Organizational Process Performance | Not ¢ 3 | 
[Basic ProjectManagement | 
[Advanced ProjectManagement | 


Integrated Project Management |} Not covered | 


Risk Management Project Monitoring & Control 


Vere | | 
Software Requirements 


Validation 
CO 
Configuration Management 
Measurement and Analysis | Project Planning 
Project Monitoring & Control 
Process and Product Quality Assurance 
Advanced Support CO 
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The Aerospace Corporation 
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Los Angeles, CA -2 
Phone (__) - 2 

Email peter.hantos aero.org 


Ors AEROSPACE 
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PARIVEDA 


CMMI® and OPM ® Collaborative Opportunities 


CMMI Technology Conference 
Denver, Colorado 
November, 2 


“OPM ” is the registered trademark of the Project Management Institute 
“CMMI” is the registered trademark of Carnegie Mellon University 


PARIVEDA The Business of IT® 


—~__ www.parivedasolutions.com 
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The state of being fully developed 
The time after physical growth has stopped 


Maturity State: An organization® degree of maturity in 
organizational project management (OPMS3) 


Maturity Level: Degree of process improvement 
across a predefined set of process areas in which all 
goals in the set are attained (CMMI) 
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fe precise about models... 


meme Features 


4 Asmall object, usually built to scale, that represents in 
detail another, often larger object 


3 An abstraction of reality 
3 worthy of imitation; fexemplary behaviord 


3 representation of something 
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4 Conceptual framework 
3, Useful references for process improvement 
4 Should allow organizations to assess themselves against 


4 Should be based on broad consensus 


A search in April, 2005 provided 268,000 
hits when ‘Maturity Model’ was used as the search item... a 
revised search in October 2007 revealed 2,480,000! 
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4 CMMI for Development Version 1.2 published by SEI 

and CMU in 2006 
A Replaces Legacy CMM6G no longer supported by SEI 
A Incorporates Four Disciplines 

- Systems Engineering 

- Software Engineering 

- Integrated Product and Process Development 

- Supplier Sourcing 


4 CMMI for Acquisition Version 1.2 published by SEI and 
CMU in 2007 


A Uses Core Model Foundation Material with Acquisition 
A Meant for acquirers instead of developers 
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feemied) Features = 


Focus on continuous 
process improvement 


Process measured 
and controlled 


Process characterized 
for the organization and 
is proactive 


Process characterized 
for projects and is often 
reactive 


Process unpredictable, 
poorly controlled and 
reactive 


Intermediate CMMI V 1.1 Topic 0803 — Page 12 
©2003 by Carnegie Mellon University 
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Project Planning 

Project Monitor Control 

Supplier Agreement Management 

Requirements Management 


Process and Product _uality 
Assurance 


Configuration Management 
Measurement and Analysis 
Integrated Project Management 
Risk Management 

Organization Process Focus 
Organization Process Definition 


Core Model Foundation Process Areas 
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process areas provides a perspective on the 


Requirements Development 
Technical Solution 

Product Integration 

Verification 

Validation 

Decision Analysis and Resolution 


Organizational Environment for 
Integration 


uantitative Project Management 
Organizational Process Performance 


Organizational Innovation and 
Deployment 


Causal Analysis and Resolution 
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Specific Goals 


3 CMMI models also include other informative materials 
A Typical Work Products 
A Sub-practices 
A References 


©2003 by Carnegie Mellon University 
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@ dard to address organizational project 


Published by PMI in 
December, 2003 

5 year project with over 800 
global volunteers 

Beta-tested by 25 companies 
in varying industries before 
release 


First standard to define 


Organizational Project Organizational 
Project Management 
Management Maturity Model 
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Por complete. Val project management 


seemed Features 


The application of 
knowledge, skills, tools 
and techniques to 
organizational and 
project activities to 
achieve the aims of an 
organization through 
projects.o 


PORTFOLIO 
MANAGEMENT 


PROGRAM | 
MANAGEMENT |\ 


OPM3® Knowledge 
Foundation, p. 5 


OPM covers three domains 


©2003 by Project Management Institute 
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Organizational 
Strategy 


ing the gap between an organization’s strategy 
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©2003 by Project Management Institute 
13 


Your complimentary | 


use period has ended. 


PDF Complete. 


serio features 


Thank you forusng PNeNtS are Best Practices, Capabilities, Outcomes 


14 


.. some are dependent on others 


Mer-yey- oY 1 57 Capability 


Rer-vey- YT 19 fo Capability 


OF-Tey-le) Ti av am < OF-Tey-Te ITA, 


15 


Your complimentary | 
use period has ended. — 


a Thank you for using _ . sata 
Complete PpF complete. OWS a best practice with three capabilities 


meted Features : 

Project Initiation Process Control: Project Initiation 
Process controls are established and executed to control 
the stability of the process. 


Stable Project Initiation Process Operations 


Implement Project Initiation Process Control Plan 


OF-Tey-1e ITA, 


Project Initiation Control Plan 
OF-Tey-1e Tia, 


Project Initiation Process 
Measurement 


Project Initiation Process 
Standardization 
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3 oer kyourrusra gorized by “PPP” and “SMCI” — more coverage of 
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saat) features 


Improve 


Control 


Measure 


Standardize 


Project Program Portfolio 


Your Organization on OPM3 Continuum 


Coverage 


Wi Organization's Relative Maturity 


Figure - Graphical Representation of an Organizations Overall Position on a Continuum of 


©2003 by Project Management Institute , . : i 
Organizational Project Management Maturity 
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that could be used to compare similar ‘guides’ 


Contents => Application —» resus | 


4 Discipline 

Sales 

Manufacturing 
Engineering 

Project Management 
Marketing 

Legal 


Jo Do Do Do Do To 


3 Domain 
A Industry 
A Company size 


3 Key question: 
A Does the content conflict? 


3 Approach or method 2 
A 


3 Extent of the change A 
A Which parts of the A 
organization? 
A How much will it change A 
the organization? 


3 Resources 
A _ Specific tools 


A People 
A Training 
A Expertise 


3 Key questions: 

A _ What do | need to do to see 
benefit from using this 
guide? 

A Where do | get help? 


Key Questions: 


What will be improved? 
How will it be sustained? 


What do | really expect to 
get out of this? 


What can | expect? 
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fem features 


3 OPMS3 covers the domain of project management 
3 CMMI covers a group of process areas covering project management 


OPM3 Domains 


CMMI Categories 


PORTFOLIO —___— 
MANAGEMENT Engineering/ 
Acquisition 


Project 
PROGRAM PROJECT Management 


MANAGEMENT MANAGEMENT 
Process 


Some of this other stuff is different! 
3 OPM3 distinguishes between project and program management and 
addresses portfolio management 


3 CMMI covers technical disciplines related to acquisition or development 
19 


hink about applying OPM3 or CMMI is similar 


Learning 


Prepare 


Assessment 


| Satisfied | 
with 
: Results , 


Silmubus for Change Condert pha. - 
Pilot Tes! 

ene = Soloflan 

Initiating Characieriz 

Gurnent & 

Ousined Sinton | 


- Perform 2A 
‘ ssessment uy) 
Diagnosing Process = 
m 
Cel Ww 
Priaritles | Develop WH 
| Agprooch ©2003 by Project Management Institute = 
| 
IDEAL® Model Establishing ~ 
Plan 
eo ae Im | a 
3 The SEI often references the IDEAL model when , 


describing how to apply CMMI 
3 The OPMS3 Cycle is part of the standard 


OPM3® Cycle 


20 


Your complimentary 
use period has ended. 

Thank you for using 
PDF Complete. 


for both models vary in terms of depth and 


feed Features 


Assessment Type OPM3 CMMI 


Entry-Level Self Assessment (SAM) SCAMPI C (or ARC Compliant Gap Analysis) 
AShort duration AComes with OPM3 standard AOrganizations can assess whatever areas of risk 


AMinimal resources AShows approximate organizational maturity on | are most important to them 
ALow level of rigor overall continuum AOutputs show which model areas are most at risk 


A151 binary (yes/no) questions AConducted by as few as one appraiser 


ACan be performed by anyone AOnly one piece of evidence (artifact or verbal 
ANo specific requirements affirmation through interview) is required 


Mid-range OPM3 ProductSuite Desk Assessment SCAMPI B 


Amore effort ARequires assessor to look at capabilities and AOrganizations can assess whatever areas of risk 
AMore rigor outcomes with process owners are most important to them but often focus on full 


AShows approximate organizational maturity if coverage of a maturity level 
rest of organization is following process AOutputs show which areas are most at risk 


AMust be performed by a certified ProductSuite AMust be conducted by a minimum of 2 team 
Assessor members and an authorized team leader 


ARequires evidence or testimony to demonstrate | AMultiple pieces of evidence (artifact or verbal 
capabilities and outcomes affirmation through interview) are required 
SCAMPI A 


AOrganizations coverage of a maturity or capability 
level 


AOutputs include a maturity or capability level 
AMust be conducted by a minimum of 4 team 
members and an authorized lead appraiser 

AMultiple pieces of evidence (artifact or ver 
affirmation through interview) are required 


SEI or PMI authorized OPM3 ProductSuite Rigorous Assessment 
rating ARequires assessor to look at capabilities and 
ASignificant effort outcomes with process owners and practitioners 
ASignificant rigor AResults in a benchmark of organizational 
project management maturity 

AMust be performed by a certified ProductSuite 
Assessor 

ARequires multiple pieces of evidence and 
testimony to demonstrate capabilities and 
outcomes 
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quires change management and both provide 
nts to maturity 


megeenae Features 


3 Change management is required for a successful usage of CMMI 
or OPM3 


Management Support 
Budget 

Resources 

Time 

Tools 


Do Do Do Do Do 


3 Organizations cané go directly to best practices over night 


Commitment! 


3 Both models provide Gtepping stonesdto higher levels of maturity 
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3 have both reported significant results 


feed Features 


3 Users of OPM3 have reported 
A Higher levels of customer satisfaction 
A Increase in their PM capabilities and aligned processes to the PMBOK® Guide 
A Visibility to how well they use their resources once they develop their portfolio 
management capabilities 
3 Users of CMMI have reported 
A Reduction in software costs either directly by streamlining processes or indirectly by 
performing less rework. 
A Productivity increases of 20% to 30% from their engineering staff 
A Quality objectives were met and the numbers of defects were reduced by using CMMI 


3 Organizations that apply both OPM3 and CMMI together can expect results as 
described above and will most likely see additional benefits such as: 


A Reviewing the overall portfolio using the best practices from OPM3 and freeing up 
resources to work on projects leading to innovation while leveraging the process areas 
from CMMI to execute those projects predictably and successfully 

A Using the project and program management guidance from OPM3 to figure out how to 
apply the planning and monitoring and controlling advice that is documented in CMMI 
and implement it at different levels of the organization 

A Achieving either OPM3 or CMMI-related objectives (e.g., cost, quality, or schedule 
improvements) faster than if they would by just considering one model alone because 
each model can provide a different perspective on the organizational opportunities 
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Situation 


3 ACMM Maturity Level 3 rated organization with a large number of Project 
Management Institute (PMI) Certified Project Management Professional& 
(PMP) seeking to achieve world class project management processes 


3 The organization was acquired so the management team also wanted to 
demonstrate their overall project management capability and take a lead in 
the new combined entity to integrate other departments and resources 


Solution 


3 Collaboratively plan for an organizational assessment with the organizational 


leadership and key team members to ensure the right resources were 
available at the right time to obtain the data needed to establish a baseline 
everyone understood 


3 Measure and baseline capabilities using the OPM3® ProductSuite 


3 Train organizational resources on critical areas where gaps were identified 
from the assessment in order to start the movement toward organizational 
process improvement and build shared vision for the future processes 


Benefits 


3 By using OPM3, company management was able to identify a root-cause 


problem of a lack of integration between the engineering and sales functions 


(lack of overall program management) 


3 The company also improved visibility to the portfolio management process 
and identified immediate opportunities for improvement 


Thank yourerusing Iement Maturity — a global telecommunications 
prove PM capabilities as a competitive advantage 


Performance Results formally reported 
through OPM3® ProductSuite 


Percentage of Best Practices Achieved 
Propect 
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| 
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Percentage of Best Practices Achieved 


Standardize 
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Improve +e 0 ‘++ 44 Measure 


Product Suite Score 
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s of the two models, the way organizations apply 
ganizations expect, support each other 


3 Key question: 


A 


Does the content 
conflict? 


3 Conclusions: 


A 


Using CMMI and OPM3, organizations can do the right projects the right way! 


CMMI and OPM3 both 
address project 
management in a 
complimentary way 


OPM3 also covers 
program and portfolio 
management 


CMMI also covers 
engineering and 
technical best practices 
Both models stress 
process improvement 
and support 


3 


Key questions: 


A 


A 


What do | need to do to 
see benefit from using 
this guide? 

Where do | get help? 


Conclusions: 


A 


CMMI and OPM3 both 
have a similar 
application style (see 
IDEAL model and 
OPMs3 cycle) 


Both require change 
management to be 
successful 


Both have certification 
processes in place to 
help find qualified 
professionals 


WN 


3 


Results 


Key Questions: 


A What will be improved? 

A How will it be 
sustained? 

A What do | really expect 
to get out of this? 

Conclusions: 

A Users have reported 
customer satisfaction, 
cost savings, and more 
productivity 

A Both models stress 
process improvement 
to sustain change 

A When you use CMMI 


and OPM3 together, 
you should have high 
expectations for results 
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4 Questions? 


45 Comments? 


4 Contact Information 
Tom Keuten 
Pariveda Solutions 
101 W. Big Beaver Road 
Suite 1400 
Troy, MI 48084 


248.224.2140 
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Copyright 2007 
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AAbout Harris RE Communications 


Origin 
AISO 9001-2000 aecieaainy 
ACM M |® Architecture 
Assessment Methods 
AOPM3¢ Improvement Methods 


AProcess Standards Comparisons 
ASynergy Concepts and Techniques 
ABenefits of Multi-Model Approaches 
AClosing Thoughts 


Ee Copyright 2007 15-Nov-07 
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cations Division 


HARRIS 


RFCD Mission 


We are the best-in-class, global supplier 
of communications-based products and 
systems for military customers. Backed 
by superior customer support, our 

reliable, fully-integrated, secure wireless 


solutions help our customers achieve 
network integrity, pervasive connectivity 
and information dominance in their 
theater of operations. 


15-Nov-07 


ACommercial Approach 

AHigh Product Mix 

ALow to High Volume 

ABuild to Forecast 

Aindustrial & Military Specifications 
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INTERNATIONAL ISO 


STANDARD 9001 : Second Edition 


Third edition 
2000-12-15 


Guidelines for Process 
[nteyration and Product 


niproventciit 


Quality management systems — 
Requirements 


Systémes de management de la qualité — Exigences 


[iary Bet nrissee 
Mike Konrad _ Organizational 
Sandy Shrom Project Management 
= | Maturity Model 


© 180 2000 . - OPMS Knowledge Foundation 
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AAbout Harris RE Communications 


Origin 
Also -2 Applicability 
ACM Mi Architecture 
Assessment Methods 
AOPM Improvement Methods 


AProcess Standards Comparisons 
ASynergy Concepts and Techniques 
ABenefits of Multi-Model Approaches 
AClosing Thoughts 
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INTERNATIONAL ISO 


STANDARD “— AA Generic Systems Process Standard 


Third edition 
2000-12-15 


ABased on 279 fFShalloStatements 
1 Not all Absolute 


Quality management systems — 
Requirements 


Systémes de management de la qualité — Exigences A S h al | State me nts G rou ped | nto 
Sections 


Alncludes Electronic and Hardcopy 
Formats 


A Application Includes, é 
1 Any Type of Organization 
1 Global 
1 Multiple Industries 


Reference number 
ISO 9001:2000(E) 


© ISO 2000 
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CONTINUAL IMPRO EMENT OFT E_  UALIT 
MANAGEMENT S STEM 


Management 
Responsibility 


Clause 


Resource Measurement, ' —_ 
Analysis and 
M t pueuse Clause fare iaal Satisfaction 
anagemen 


Clause 


Service/Product Service 
Requirements Realization Product 


ictivities 


Copyright 2007 15-Nov-07 


Your complimentary 
use period has ended. 


= Complete ""eeuaaaa 
ora tn a 
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A Certification is known in some countries as registration. 


A An independent, external body audits the organization's 
management system and verifies that it conforms to 
requirements specified in the standard (ISO 9001-2000). 


Initial Transition 

- Optional - On going 
pre-assessment surveillance 

- Registration - Upgrading the 
assessment existing registration 
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| Improvement 


femme Features 


A Continual Improvement is both a 9001-2000 Quality 
Management Principle and a Requirement 


Continually Improve Effectiveness of the System 
Corrective Action 
Preventive Action 


A 1SO 9004-2000 Guidelines Recommended Methods 
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Financial Measurement 
Measurement of Process Performance throughout the Organization 
External Measurement, such as Benchmarking and Third-party Evaluation 


Assessment of the Perceptions of Customers and Other Interested 
Parties of Performance of Products Provided 


Measurement of Other Success Factors Identified by Management. 


Assessment of the Satisfaction of Customers, People in the Organization 
and Other Interested Parties 
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CMMI 
Second Edition 


Guidelines for Process 
Integration and Product 
lmpeavenicnt 


| Mary Beth Chrissis 
Mike Konrad 


Sandy Shirin 


Model HARRIS 


AA Model-Based Process Standard 


AConsists of Generic and Specific 
Practices (437) 


APractices Grouped into Process 
Areas 


Alncludes Electronic and Hardcopy 
Formats 


A Application Includes, é 


1 Product Development or Service 
1 Global 
1 Multiple Industries 
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Maturity Levels Process Areas (PA) 
(2 PAs) Organizational Innovation and Deployment (OID) 
Continuous Process foyeytlralate Causal Analysis and Resolution (CAR) 
Improvement () 
2 PAs itati 
( s) ; uantitatively Organizational Process Performance (OPP) 
Quantitative WE Tar=Ke[=x6) uantitative Project Management (_ PM) 
Management () 
( PAs) BY =yilaT=xe| Requirements Development (RD) 
Technical Solution (TS) 
Process 
Standardization - Product Integration (Pl) 
erification (_ ER) 
alidation (AL) 
Organizational Process Focus (OPF) 
Organizational Process Definition (OPD) IPPD 
Organization Training (OT) 
Integrated Project Management (IPM) IPPD 
Risk Management(RSKM) 
Decision Analysis and Resolution (DAR) 
( PAs) Requirements Management (RE M) 
F P Project Planning (PP) 
Basic Project 
aden ant Project Monitoring and Control (PMC) 
Supplier Agreement Management (SAM) 
Measurement and Analysis (M_ A) 
Process and Product uality Assurance (PP A) 
Configuration Management (CM) 
Taliees) Ad hoc, chaotic processes 
() 
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g - Goals & Practices 


SG 1 Establish Estimates 


SP 1.1 
SP 1.2 
oP Is 
SP 1.4 


SG 2 Develop a Project Plan 


SP 2.1 
SP 2.2 
SP 2.3 
SP 2.4 
SP 2.5 
SP 2.6 
SP 2.7 


Estimate the Scope of the Project 

Establish Estimates of Work Product and Task Attributes 
Define Project Life Cycle 

Determine Estimates of Effort and Cost 


Establish the Budget and Schedule 
Identify Project Risks 

Plan for Data Management 

Plan for Project Resources 

Plan for Needed Knowledge and Skills 
Plan Stakeholder Involvement 
Establish the Project Plan 


Bullapso 


SG 3 Obtain Commitment to the Plan 


SP 3.1 
SP 3.2 
SP 3.3 
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Review Plans that Affect the Project 
Reconcile Work and Resource Levels 
Obtain Plan Commitment 
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fee Features 
Requirement SCAMPIA SCAMPI B SCAMPI C 


Types of Objective Documents Documents Documents 
Evidence Gathered and Interviews | and Interviews | or Interviews 


Ratings Generated Goal Ratings No Ratings No Ratings 
Required Allowed Allowed 


Organizational Unit Required Not Required Not Required 
eaten 


a Team Leader | SCAMPIA SCAMPI B — B 
Requirements Lead Appraiser | and C Team and C Team 
Leader Leader 


SCAMPI = Standard CMMI® Appraisal Method for Process Improvement 


si Copyright 2007 15-Nov-07 
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A Continual improvement is the focus of CMMI®. 


1 CMMI® is a froadmapofor improvement through its maturity 
levels. 


| All process areas have improvement practices. 


1 Improvement approach is gradual. 


A CMMI® provides improvement guidelines for first time 
adopters and organizations transitioning from other 
process standards. 


A CMMI® based appraisals are used to guide process 
improvement. 
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» Standard 


AA Model-Based Process Standard 
ABased on 586 Best Practices 


ABest Practices are Broken Down 
into a Series of Capabilities (or 
Steps) 


AAvailable CD-ROM and On-Line 


AApplication Includes,é 
1 Any Type of Organization 
1 Global 
1 Almost any Industry 
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Organizational Project Management, its 
maturity, relative Best Practices, and 
how to use OPM3® 


[mprovement, 
Development of an a 


improvement plan 7 J e —\ 
.~ Assessment” ” 


Comparison of current 
maturity state with the 
model 


Sequence of developing 
Capabilities aggregating 
to Best Practices 


Includes methods for 
evaluating Best Practices 
and Capabilities 
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Capability 1010.020 

The Organization assembles, 
develops, purchases or 
otherwise acquires a Project 
initiation process. 
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Capability BG Outcome 


Best Practice 1010 


Project Initiation Process Standards are Established 


Outcome 1010.020.10 

Project Initiation Process is 
documented and 
communicated to all necessary 
stakeholders 


Copyright 2007 


Key Performance Indicator 
A Documented 

Project Initiation Process 
Exists 


15-Nov-07 


e Structure - 2 


BP 3060 - Select Projects Based on 
Organizational Best Interests 


f 


Align portfolio org. objectives and strategy 


I 


Prioritize projects based on strategic business objectives 


t 


Prioritize projects according to short-term org. objectives 


i 


Select projects supporting sponsor best interests 
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gorizations 


“,..groupings to provide structure and a framework for the 
OPM3® Model so that the relationship between Best Practices 
and Capabilities could be better understood.” 

OPM3® Knowledge Foundation 


Domains: 


Improvement Stages: 


Project Management 
Processes: 
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Portfolio, Program or Project (PPP) 


Standardize, Measure, Control or 
Continuously Improve (SMCI) 


Initiate, Plan, Execute, Control, and 
Close 


Copyright 2007 15-Nov-07 


Step 1. Prepare for Assessment 
Know the model. 


Step 2. Perform Assessment 


Use Self and Comprehensive 
Assessments initially. 


Step 3. Plan for Improvements 
Rank the improvement needs. 


Step 4. Implement Improvements 
Monitor and control change (s). 


Step 5. Repeat the Process 
Return to Steps or 
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suring Maturity HARRIS 


saermieed) Features 


AOrganizational Project Management Maturity 
Increases along a Continuum 


1 OPM2® provides multiple perspectives 
for assessing maturity 


Portfolio 
1) Along the SMCI 
2) Along the PPP Prograti 
3) Progression of capabilities 
leading to best practices 
Project 


A) Initiating, planning, executing, 
controlling and closing 


Stendardize Mensure Control Continuously 
Imiprowe 
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AClosing Thoughts 
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axe) aaars li Model-Based Model-Based Standard 
lp EIEEY Appraisal Registration 
Method 


VEESticuicniaee) Maturity Continuum Maturity Capability One Set of “Shall 
Level Requirements 


Institutionalization and 
Implementation 
Guidance 


eee! Narrow Narrow Broad 
Application . : 
(Project Management) | (Product Development) | (Cross Functional) 


No Implementation 
Guidance 


Taale)(-Verlclalcliteam Implementation 
equi Te F-Tater= Guidance 
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AClosing Thoughts 
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ode! Environment HARRIS 


eee Features 


ACore Organizational Process Requirements 
1 Built on ISO 9001-2000 and CMMI® 


A Process Requirements are focus of change 
A Not the model 


A Audits, Appraisals, Lean, Six Sigma, etc., are 
improvement methods 


UALIT MANAGEMENT S STEM 


MI FRAMEWORK 


a CMMI ML 
Iso. 


1 Unified Process Improvement Efforts 


AAll models support process improvement for 
groups and interfaces among groups 


A All models have focus and non-focus areas 


AAIll models have best practices 


1 Institutionalization / Standardization of Practices 


A Strongest in CMMI® with Generic Practices 


ML = Maturity Level 
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d Benefits 


Famed Features 


A Planned Multi-Model Approaches Unifies and Disciplines Diverse 
Process Improvement Efforts 


| Promotes and aligns Leadership understanding and commitment 

1 Avoids redundancies and overlaps in process improvement ($$$) 
A Encourages coordination and resource optimization 

1 Helps to eliminate self-optimizing silos and process overlaps 
A Encourages efficient subprocess ordering and interfaces 

1 One model enables another model 

l Helps to capture the overall business process framework 

A Assessments Provide Answers to Organizational and Business 
Process Maturity 
1 Does the process contribute to value-added work flows? 
1 Does the process contribute to organizational/business objectives? 
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4 Pillars of Wisdom 


=" “Don't pay for the same real 
estate twice.” 


<= “Dont take down a fence until 
you know why it was put up.” 


Ss “| ..build it and he will come.” 


‘= “Change before you have to.” 
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Mark Scott is the Senior Manager for Process, Productivity and 
Improvement at Harris Corporationé RF Communications Division, 
Rochester, N.Y. He has 10 years practical experience leading and 
assessing model-based process improvements for project/program 
management, engineering, quality assurance, and supply chain 
organizations. Scott currently leads RF Communications6CMMI® and 
OPM32? initiatives, and is responsible for integrating these process 
frameworks into the division® ISO 9001-2000 Quality Management System 
(QMS). Along with a Master Degree in Organizational Management, Scott 
is a PMI licensed OPM3® assessor, a Six Sigma Black Belt, and a Harris 
candidate for the Carnegie Mellon University, Software Engineering 
Institute, CMMI® lead appraiser program. 


Email: Mark.Scott@harris.com geet” OP a., 
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wa\e” 4 4 tl 

Mark N. Scott 


= Copyright 2007 15-Nov-07 


Your complimentary qi 
use period has ended. 


Complete "aa 


ag Features 


Tom Cory and John Lowell, Manufacturing Best Practices Webinar: Business Process Improvement”, 
Session 1, December 5, 2006 


Kathleen Huneycutt, FOPM3® a Proven Protocol for Performance Improvement”, Presentation, 2006. 


Lesle Hill, FfProject Management Body of Knowledge (PMBOK®) and the Capability Maturity Model 
Integration (CMMI®)”, Presentation, Rochester, N.Y., 2004. 


Kirwan, K., Urs, A., Sassenburg, H. and Andre Heijstek, A Unified Process Improvement Approach for 
Mult-Model Improvement Environments, a Carnegie Mellon University, Software Engineering Institute 
Publication, Pittsburgh, PA. 2006 


Matufelija, B. and H. Stromberg, Systemic Process Improvement Using ISO 9001:2000 and CMMI®, 
Boston, MA.: Artech House, 2003. 


Praxiom Research Group Limited, What’s New: !SO 9001 2000 versus ISO 9001 1994”, December 12, 
2006. 


Jim Sloane and Lisa M. Kruszewski, FOrganizational Management Maturity Model”, Presentation, 2003. 
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Winner IEEE Software Process Achievement Award 
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Topics 


> Issues 
Quality and Schedule 
Rational Management and Commitment 
Insanity and Malpractice 
> Three Improvement Perspectives 
Organization - CMM/CMMI 
Individual 1 PSP 
Team 1 TSP 
> Lessons Learned 
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in oa s More Imp ortant Than 
Wrolatereltiis 


Mn todayG software marketplace, the 
principal focus is on cost, schedule, and 
function; quality is lost in the noise. This is 
unfortunate since poor quality performance 
is the root cause of most software cost and 
schedule problems.0 


Watts Humphrey 
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INauvuar ivianagement - Developers 


> When pressed for early deliveries, the 
responsible team members say 


rl understand your requirements, I willdo my 


utmost to meet it, but until I make a plan, I can 
not responsibly commit to a dateO 
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nauvuuar Management - Managers 


> When pressed for early deliveries, the 
responsible managers say 


rl trust you to create an aggressive and realistic 
plan, I will review the plan, but I will not 
commit you to a date that you can not meetO 
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iNauvuar Management - Principles 


> Set challenging goals 

> Get the facts 

> Use facts and data 

> Anticipate and address problems 


ais 


Advanced Information Services Inc. OWN WAN Or 


Your complimentary 
use period has ended. 


ee Complete "a 


weesenity or Malpractice? 


Unlimited Pages and expe 


Insanity 


Doing the same thing over and over and 
expecting a different result 


Malpractice 


An organization which does not have a 


top-management-sponsored 


continuous improvement initiative in place 
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Level Key Process Areas (KPA) 


Optimizing Continuous process Defect prevention 
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> Getting management attention 
> Maintaining long-term improvement focus 


> Guiding the improvement work 
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GM 


> Average number of days late in meeting milestones declined from over 
50 days to fewer than 10 following organization focus on CMMI 


EE vo Days I. ate| 


CY 200k 6 SA =6Part 2002 


General Motors Presentation, SEPG, Boston, MA, 2003 
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OT cAdM Level 1 | CAM Level 2 | Cl4td Level 3 | CMM Level 4 | CAM Level 5 
mOefectsKLOC| 75 | 624 | 473 | 228) | 4.05 


The TSP in Practice, SEI Technical Report, September 2003 
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Uae to mdvance from MLI to MLS 


22 months 19 months 25 months 13 months 
ML1——>+}> ML 2 ——-} ML 3————>}-ML 4 —>}-ML 5 


Source: Software Engineering Institute 
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> No simple model could precisely measure process 
maturity and complex models are not useful in 
guiding improvement 


> CMM consciously focused on what organization 
should do, not on how they should do it 


> The teamwork practices and personal disciplines 
required for quality software work are almost 
entirely issues of how, and not just what 


> Because engineers will not change the way they 
work without very specific guidance, the CMM 
does not change engineering behavior 
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> The need is not for lots of process data but for 
engineers who gather and use that data 


> What would happen if software professionals used 
sound engineering practices? 
{imas UOT Ce Comrevee Ko) EConniore me Core-TU (cre me) E-vels 
| gathered and used historical data 
| measured and managed quality 
1 analyzed and improved their processes 


> The need 1s for a Level 5 Process at the individual 
level 
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From Project To Project 


NYou can not stand still, so you should treat 
every project as a way to build talent rather 
than merely treating your talent as a way to 
build projectsO 


Watts Humphrey 
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Personal Software Process - 1 


Team Software 
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PSP sore Tale) up PSP Requirements 


oa 7eq [tome (e\1Z-1(0) )ant-1a] Configuration management 
yenceoversem”" | methods to larger 
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(overs (-Wa-WI-\ WES Design templates 
Design reviews 


PSP PSP . size, resource, and schedule plans 


F “eee Task planning 
Size estimating = 
Test report Yor a(-re UI (We) E-lalaT ale) 


PSP . ee 
PSP Coding standard establishing a measured performance 


Current process Size measurement F 
Time recording Process improvement etstsys)| Wats 
Defect recording proposal (PIP) 

Defect type standard 
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> At the end of the PSP training, developers 
know how to: 
Consistently gather size, time, and defect data 
Make commitments based on historical data 


Analyze personal data to answer questions 
| Where am I spending my time? 
| What are my common defects? 
| Where do I inject the defects? 


| What goals do I need to set to improve? 
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User Acceptance Defects KLOC 


Defects KLOC 


Project Start Date 
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PSP Problems 


> To do quality work, engineers need a detailed plan 
and a defined process 


> Without the process, they cannot make detailed 
plans, take consistent measurements, or track their 
work against the plan 


> However, when engineers have a project to 
deliver, they are rarely willing to take the time to 
define a complex process, even when they know 
how 
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> Need a mechanism to guide teams through 
defining their processes and making 
complete, precise, and detailed plans 


> Need a vehicle to help organizations 
capitalize on the potential benefits of 
disciplined teamwork 
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~ ll am Improvement 
Jelled Teams 


rhe speed with which organizations form 
and deploy teams is the single most 
important factor in determining their 
competitive successO 


Melled teams are the most powerful tool ever 
devised for doing challenging workO 


> Watts Humphrey 
ais 


——E—E———————————————————————————— es 
Advanced Information Services Inc. OW NWA Or 


Your complimentary 
use period has ended. 


ee Complete aaa 
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Self-directed Teams 


> Characteristics of self-directed teams 
| Sense of membership and belonging 
| Commitment to a common team goal 
| Ownership of the process and plan 


| The skill to make a plan, the conviction to 
defend it, and the discipline to follow it 
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> With PSP training, developers know how to plan, 
schedule, and track their work 
> TSP teams use these PSP-learned methods to make 
detailed plans 
| Tasks are no more than 10 task hours each 


| Task time is recorded daily 


| EV is measured weekly 
> You can tell project status to within 10 task hours 
> TSP teams regularly report their status 
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Project Tracking Issues - 2 


> Project schedules slip a day at a time 


> If you cannot precisely measure project status, you 
will not know where projects stand 


> Without such knowledge, you cannot address 
schedule problems in time to fix them 
> With the TSP, you can 


1 closely monitor team performance 
| address problems in time 


1 consistently meet schedules 
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Average Task ours Per Team Member 
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Table 1: P-3C Process Improvement Results 


Source Lines of 
Code (SLOC) 


Productivity 
(SLOC/hr) 


Development de- na na 
fects 


Plan Release Date prone? 12/4/2001 1/26/2004 


Actual Release Date 5/29/2001 2/5/2004 


Ps | i Ss Source: SEI Technical Report Case Study: Accelerating Process Improvement by Integrating the TSP and CMMI 
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Timeline to Advance to ML4 


NW/FeV ao a 40] 0 16) began CMM-based improvement effort 
(OJoi (e) 01-1 a40) 016) began PSP/TSP introduction 

January 2001 FelulaloialcYoln i] esau ie) eam (cre 100 

WiFe Waa 0 6M reached Maturity Level 2 

June 2002 launched second TSP team 


September 2002 reached Maturity Level 4 (SW-CMM) 
Source: SEI Technical Report Case Study: Accelerating Process Improvement by Integrating the TSP and CMMI 
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Level 3 
Organization develops 


Organization standard processes 


Level 4 
Project mgrs. establish Development process 
discipline & stability managed statistically 


Level 1 
Ad Hoc processes. 


Individual inconsistent results 


Source: fFrom MCC to CMM64, Dr. Bill Curtis, DC SPIN, April 2006 
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PIP# : 
Written By: 
Date : Author(s): * Project: ae 


Process Name : FL, Key Process Area: * 


Luprovement Description : 
F 


J 
Duprovement Benefits (Check One): 

© Document Improvement C Reduced Cycle Time 
© Improved Quality © Reduced Risk 


Benefits Description (Quantify Where Possible) : 
(Attach files if needed) 
o 

Jj 


Attach the PIP Pilot Report here (if applicable): ‘ 


w SEPG Evaluation 
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~~ssons Learned - | 


> While models are useful to indicate where 
improvements are needed, only committed people 
can make the improvements 
A supportive management environment that 
rewards disciplined behavior is absolutely 
essential 


Timely feedback on the status and disposition of 
the PIPs is important to sustain the PIP 
mechanism and feeling of empowerment 


Do not need to wait till level 5 to start 
implementing process change management 
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While CMM is necessary as an organizational capability 
improvement model, it is not sufficient to change 
engineering behavior; the PSP provides the detailed Mhow 
toOfor improvement at the individual level 


The TSP provides the management framework for 
continuously improving self directed teams. The PIP 
mechanism is key for team ownership of the project@ 
process and commitment to improve 


CMM, TSP, and PSP all three are needed for an 
integrated approach to model based improvement at the 
organization, team, and individual levels without the risk 
of sub-optimization 
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October 22-25 


> Study of 3700 findings from assessments! More than 
half negative 


yeas we beds er-lorcloyOUIAvar-baCe mont-TRelminraelOmeCOlMmcautclu-balkcioms)nescan:ionl 
SUCCESS 


> Programs fail because Nwe donG start them right, we 
don@ manage them righto 


> Developers often at lower maturity level than 
organization 


CMM, TSP, PSP 1 Why we need all three 
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arks and Service Marks 


> The following are service marks of Carnegie Mellon University. 
CMMI 
Team Software Process™ 
i is] 
Personal Software Process®“! 
PSP>™ 


> The following are registered trademarks of Carnegie Mellon 
University. 

Capability Maturity Model® 

OY AY 

Capability Maturity Model® Integration 

CMMI® 

CERT® 
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Girish Seshagiri 


Advanced Information Services Inc. 


(703) 286 0781 
Email: girishs @ advinfo.net 


Website: www.advinfo.net 
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7“ Annual 
CMMF Technology 
Conference 


Formal Process Definition 
with Industry Standards 


Armstrong Process Group, Inc. 
www.aprocessgroup.com 


Copyright © 1998-2007, Armstrong Process Group, Inc., All rights reserved APG 
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Formalize an organization's process architecture using industry 
standards 
e Software and Systems Process Engineering Metamodel 2.0 GPEM) 
e Discuss different compliance levels of formal process definition 
Build a process asset library (PAL) using method content (tasks, 
roles, and work products) 
Configure base method content into different configurations for 
different lifecycle models 
e Enable enterprise process integration 
e Make actionable work breakdown structures which precisely 
represent the organization defined process 
e Eclipse Process Framework (EPF) overview 


e Briefly demonstrate an open source tool (EPF Composer) for 
capturing, tailoring, and publishing process definitions 
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ECOMplete nC mae 


ent Teams Are Facing Today 


fiend Features 


Project plan Article on serialized Website with 
templates java beans Configuration mgmt 
guidelines 
Book on J2EE 


Lessons learnt from 
previous project and iteration 


JUnit user guide 


Latest research on 
effectiveness of pair programming 


Wiki on agile 
development 


Knowledge base on 


aoe . Corporate guidelines 
managing iterative development 


on compliance 


e No common language or terminology between processes - redundancy and 
inconsistencies 

e Knowledge cannot easily be customized for different projects or new best practices 

e No central community or communication framework to facilitate convergence of 

best practices across domains 
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3 Complete se DDE Complete, 


tions Are Facing Today 


fem Features 


e THe multiple, concurrent process lifecycles 


e Enterprise architecture, portfolio management, solution 
delivery, service management, governance 


e Difficult to integrate process lifecycles 
e Unclear roles and responsibilities 


e Challenges acquiring new resources (whether 
employees or contractors) and rapidly orienting them to 
internal processes 


e Must be compliant with internal and external processes 
for meeting regulatory and quality requirements 
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ecture and Integration 


)Features 


Processes arranged in a “list” fashion — no easy way to navigate and use 
processes - “difficult to follow the trail of breadcrumbs!” 


Development Project 1 
Development Project 2 


Sustainment Project 1 


Development Lifecycle || Sustainment Lifecycle 
Sustainment Project 2 
2 : 0 


a ee 
=— a" 


pont o_olo so 


Project Organization 
Management Measurement 
Repository Repository 
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see) Features : t 0 n O bj e ct ves 


e Create a defined process that enables achievement of cost 
of quality and cost of production goals 


e Improve ability to acquire appropriate resources via clearer 
role definition 
e Identify lifecycle integration touchpoints 
e And outsourced vendor handoffs 
e Enable new project resources (either new employees or 


new contractors) to get up to speed very quickly by 
embracing industry standards 


e Enable useful asset management through standard work 
products and integrated tool usage 


e Improve employee satisfaction via clearer roles and 
increased team collaboration 
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lustry - Process Management 


eee Features 


ZU0/ 


Three-Ring Binders Websites Model-Driven 


¢ Demonstrated * More accessible by ¢ Access to industry 
Organization commitment __ practitioners standard frameworks 
¢ Often became shelfware = « Often difficult to navigate —« Integration of multiple 
and maintain lifecycles 
¢ Formal process asset 


management 
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Standardize representation and 
manage libraries of reusable 
Method Content 


Develop and manage Processes 
for performing projects 


Content on agile {| JUnit user Lessons learnt from 
development == guidance previous project and 
iteration 


, | y 
ontent on 

: an Standard or 
managing on J2EE reference processes 


iterative development Al Corporate 
Guidance on il Configuration guidelines = Project plan 
mgmt on compliance »<'— templates 


serialized java beans ae 
guidelines 


NN fF 


Configure a cohesive process framework 
customized for my project needs 


v 


Create project plan templates for 
Enactment of process in the context of my project 


Process assets 
patterns 
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ems Process Engineering Metamodel 


SPEM 2.0 was adopted by the Object Management Group 
(OMG) in December 2006 — SPEM 1.0 around since 2002 
e Provides necessary concepts for modeling, documenting, 
presenting, managing, interchanging, and enacting 
development methods and processes 
e Provides standardized representation and managed libraries of 
reusable method content 
e Supports systematic develooment, management, and growth of 
development processes 
e Supports deployment of method content and process needed by 
defining configurations of processes and method content 
Supports enactment of process for development projects 
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ant — Guidance 


e Can be associated with any process model element to 
provide more detailed information about the element to 
the practitioner 

e Can standalone — does not have to be associated 


e Most often associated with activities and work products 


e SPEM comes with a set of built-in guidance types 
e Checklist 
e Template 
e Example 

e Tool mentor 

e Guideline 
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Roles responsible for work 
products 
e Each work product 
responsibility of single role 
Process roles perform tasks 
e Each activity only performed 
by single role 
Work products used as 
inputs to tasks and tasks 
from activities 


“Somebody does something 
that changes something” 


APG 
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fPeatures 


[Early Elaboration teration) [ception teration (Optional) 


ISS 


«spem2Default_AchlitionalPerformer» “sPem2Default_Pri 
Define = Candidate Architecture 


Ferjorm Architectural Synthesis 


e © Use Case Analysis 


| Create the Analysis Use-Case Realization | 


«spem2Default_OptionalInput» ; Vv 
| Find Analysis Classes fr 


‘om Use-Case Behavior | «spem2Default_ManclatoryOutput» 
Analysis Moclel é 


Analysis Model ~ Analysis eer | 


| Distribute Behavior to Analysis Classes | pnalvals Woget 
Se pe /Use Case Realization 


Refine the: srebRecture =\5 hecerse sioae «gpem2Defauilt_MandatoryOutpet® 
| Describe Responsibilities | 
Bg Bs Bs Use Case en 


Design Components Design the Databace Design Services 


Analyze Behavior 


Use Case Realization 


| Describe Attributes and Associations | 


. 


AShow process behavior and team collaboration with UML activity diagrams 
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£8 requirements_management 


£8 requirements _with_use 


Systeny_Analyst\RM 


: f Use_Case_Moclel 
«responsible aes RM_Attributes RM_Traceability 


Use_Case_Moclel = 


RM_Attributes RM_Traceability 


Deselecting package 
lice)pamexe)avile [elec lire)al 
removes it from 
practice spaceo 


N=Yercole lave ma nevalilelelesle)(=xess) 41 Swale msy=) er=Ue-ltem efor <-le [=< 
AMerges elements together during publishing 
AThree kinds of variability: contributes, replace, extends 
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ocess Components 


aed Features 


My Process O 


:Analysis ; 
Requirements saneiysis Analysis 5 
E | OF > F Analysis 
Requirements 
:Development aa Application 
! O 
:Design |_| 
|__5)] Kon Application 
| 
Design Gates 


AReusable process components — “black boxes” 

Alntegrated with one another via input and output work products 
ACan substitute one component with another component (i.e. for 
different techniques or technology platforms) as long as input 
and outputs are the same 
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i > ee 
+) Plan an Iteration extends 'Plan an Iteration, msf-agile’ 
ARevisiti ale) design =] YY Design Work extends ‘Design Work, msf-agile’ 
+ Create 3 Scenario extends Create a Scenario, msf-agile’ 
work based on same =| Create Solution Architecture extends Create Solution Architecture, msf-agile’ 


WiaterVanviialem ey-liccyaal CQ Partition the System 
Ug Determine Interfaces 
A re 0g Develop Threat Mode! 
Dare lanl (omiialdiale me) ES Develop Performance Model! 
patterns increases m4 Create edinactoral Hotelfpe 
: : 1 sy Create Infrastructure Architecture 
maintainability =| @& Intermediate Project Phase 
co e Plan an Iteration extends 'Plan an Iteration, msf-agile’ 
ACh FeValeless) Tal le) atterns =] @S Design Work extends ‘Design Work, msf-agile’ 
: ] BS Create a Scenario extends Create a Scenario, msf-agile’ 
require zero updates =| BY create Solution Architecture extends ‘Create Solution Architecture, msF-agile’ 
Cg Partition the System 
Ug Determine Interfaces 
0g Develop Threat Mode! 
Dg} Develop Performance Mode! 
0g} Create Architectural Prototype 
0g Create Infrastructure Architecture 
+ @) Late Project Phase 


Many processes today are iterative, i.e. they revisit the same work. 
Process Patterns provide reuse, but not every iteration has the same focus.... 


Software Engineering Economics: Why, What and How? AP € 
Copyright © 1998-2007, Armstrong Process Group, Inc., All rights reserved 


ASuppressed two tasks 
AThey will not be 
performed 


AContributed one new 
task 
7iNeyer-\e-telelittelanre 
dynamically linked 
pattern 


AReplaced whole 
activity with new 
breakdown 

AOther pattern parts 
such as rCreate 
Scenarioostays as is 


=| 


es When Applying Patterns 


[Eregentaton name _tndex | Modelifg__| pe 


.h ar rocece 


5 Early Prcieek ae 


Plan an Iteration 
: ; Design Work 


a BS 


a 


Create a Scenario 

Create Solution Architecture 

Dg. Partition the System 

Ug} Determine Interfaces 

Dg Develop Threat Mode! 

ES Develop Performance Mode! 

Ug Create Architectural Prototype 
0g Create Infrastructure Architecture 
Ug} Review Objectives 


=| @& Intermediate Project Phase 


f Plan an Iteration 
=) Design Work 


: 
= 


Create a Scenario 

Create Solution Architecture 

Ug Determine Interfaces 

Ug} Develop Threat Model 

Cg Create Infrastructure Architecture 


@& Late Project Phase 


1 


extends 'Plan an Iteration, msf-agile’ 
extends 'Design Work, msf-agile’ 
extends Create a Scenario, msf-aqile’ 


locally contributes to ‘Create Solution Architectur... 


extends 'Plan an Iteration, msf-agile’ 
extends ‘Design Work, msf-agile’ 
extends Create a Scenario, msf-aqile” 


locally replaces ‘Create Solution Architecture, msf... 


Task Descriptor 
Task Descriptor 


Activity 

Task Descriptor 
Task Descriptor 
Task Descriptor 
Phase 


ATailored process still using dynamically linked patterns parts (green lines) 
Alt only stores its local changes 
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and Work Products Activity Diagram 


S> bo 


Create Analyze the Refine the Initiate Project 
O performs Architectural Archtecture Architecture = 
Proof-of- 
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Architect 
el Manage Iteraton 
Architecture ore 
roof-of- 
Concept Manage Requirements 
Determine Architectural Feasibility 


Assess and Plan Next Iteration 
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Work Breakdown 
Structures 


Presentation Name 
=| Hid Basic Unified Process 
fife!) Inception Phase - Iteration n 
=| G3) Manage Iteration 
®> Initiate Iteration 
®@:> Conduct daily meetin 
=| GB) Initiate Project 
@> Define Vision 
®> Plan the Project 
=| GS) Manage Requirements 
@> Describe Requiremen 
®> Find and Outline Actc 10 
®> Detail Use Cases il 
=) @ Determine ArchitecturalF 12 
® Analyze the Architeci 13 
@: Create Architectural; 14 
=| GB) Assess and Alan NextIte 15 
@> Assess Results 16 
®> Prioritize Work 17 
®> Plan Next Iteration 18 


ovo cua ps 


®&> Refine Project Plan 19 
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me Features 


Audience 


e Large development organizations 
and large scale process and 
method library tool providers 


Needs 


e Those who need all capabilities 
e Provide support for large scale 


7... - libraries of textual method content 
—_ 8 i ote and reusable process components 
| = ; © Focus is on managing many — 
! ' processes for complex multi-tiered 
\ ; | Menage}; organizations that manage 
, ‘| _ interrelated processes 


| Method plugin 
| e Process behavior 
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with Behavior and Content 


e Audience 
e Modeling focused organizations 
4 e Needs 
Oe e Work on one process at a time 


a e Create work breakdown structures 
and/or workflow diagrams 


e Who do not require 
Process e Reusable method content 


|| 
Process Ue iees ° Variability and process 
Behavior configuration 


e Provides backwards compatibility for 
cia SPEM 1.x implementations 


seem Features 


ee SPEM Packages 
‘Cow _ | e Managed content 
3 ' e@ Process behavior 


bee ee ee ee eee ee Soy 
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e Audience 


e Method documenter and book 
authors, organizational knowledge 
base provider 


e Needs 
= 
Method e Those that manage documentation 
of descriptions of development 
methods, techniques, and best 
practices 


[4 
e Who do not require 
Behavior 
! } e Formal process models 


SETTET ce e Variability and process 


eolen configuration 


Process 
Structure 


e Managed content 
e Process behavior 
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Emme Features 


e Audience 
e Modeling focused organizations 
e Needs 


e Create work breakdown structures 
and/or workflow diagrams 


e Minimal supplemental narrative 
e Who do not require 


[ | 
Process 
Behavior e Natural language guidance 


Managed e Reusable method content 


ore e Variability and process 


configuration 


: SPEM Packages 
' e Process behavior 
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aed Features 


e Audience 


e Work breakdown structure focused 
organizations 


e Needs 


e Create work breakdown structures 
only 


e Provide natural language guidance 
e Who do not require 
=a 
Snes e Formal process models 
e Reusable method content 
e Variability and process 
configuration 


| SPEM Packages 
e Process structure 


e Managed content 
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Complete ™2aaaas 


s Framework 


e New Eclipse Foundation project 7 elie 5 
e Provide an extensible framework and . : 
exemplary tools and content for software 

process engineering 
e Started in January 2006 with 16 committers 
e Have had many face-to-face meetings in US and Europe 


e Many special interest groups (SIG) aligned with process 
content and tools 


e Completed first official, public release in October 2006 
e Visit project website at www.eclipse.org/ept 
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fee Features 


e Extensible framework 


e Meta-model based on OMG’s Software and Systems 
Process Engineering Metamodel (SPEM) 


e Core extensible process tooling framework 

e Exemplary and extensible tools 
e Method and process authoring 
e Library management and content extensibility 
e Configuring and publishing 

e Exemplary and extensible process content 
e lterative, agile, and incremental development 


e Applicable to a broad set of development platforms and 
applications 
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Click Here to upgradete 


Unlimited Pages and Expanmas 


Free Process 


Content inhonce 


Content 
Plug-ins 


Plug-ins 


Commercial 
Process 


Content 


Agile/XP TOGAF Plug-ins 


(Open Unified ADM 
Process) 


OpenUP 
TOOLING sd leat 
=> 4165-9] 0) (=a OF FE} (0) 111 74:)0) (=e (=> dL 0) (2) Ui dalolalale Pmecelelcialiarep) 


Common Language & Vocabulary META MODEL (based on SPEM) 


Open Source Development ECLIPSE 
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ea Features 


Produces: 
- Base methods 


- Realistic consistency 
)- Viable governance 
'- Improved ROI 


Management NN / 


[4 Desires: 


eS -» Performs: 
a S ze aor =| Ya - Tailorin 
=“ Provides: | IN / - Simplicity "ly Pabising 
7 - Traini - Templates ” IN 
hie haInng Za» i Support 
esa — _ - Examples 


- Gui Process ~ [raining 
: - Mentoring . Professional ~ Culdance 
- Adoption services 


~ N Coach 

Service i 

Provider x Wants to: 
ae Build tools 


a At WwW, - Sell tools 


- Sell services 


Needs: 
- Teachable material 


- Teach process development 
- Use in student projects 
- Bring research to mainstream 


Tool Provider 


Academia 
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en Unified Process (OpenUP) 


e A process framework united by a set of core principles 

e Application of an iterative lifecycle that mitigates risk 
early and often, and shows results early and often 

e Focus on the collaboration within a development team 
including the product stakeholders to maximize results 

e Management of requirements in a form that 
represents stakeholder value and drives the 
development process 


e Cognizance of architecture as a means to increase 
quality and technical understandability 
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- Conti ee Compete 
An iterative software development process 
Minimal — only fundamental content is included 


Complete — can be manifested as an entire process to build a 
system 


Extensible — can be used as a foundation on which process 


content can be added or tailored as needed 


“The least amount of process for every project; 
but not all of the process for all projects” 


= GT var saconson 


Aelele OGUE Be Capgemini 


CONSULTING .TECHNOLOGY. OUTSOURCING 
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3 Compa a 


Demonstration 


Eee Features 


e View method plug-in (OQpenUP/Basic) 
e View method content (role, task, work product) 
e View process configuration 

e Create new method plug-in 

e Extend existing method content 
e Add new method content 

e Publish process 

e View published website 

e Export work breakdown structure 
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feed Features 


Questions/Discussion 


Thanks for your attention and participation! 


Armstrong Process Group, Inc. 
www.aprocessgroup.com 
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Mrs. Doubtfire Answers Your 
Questions about Process 
Improvement 


CMMI Technology Conference & User Group 
12-15 November 2007 
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Background 


- Is your organization acting “childish” 
about process improvements? 


- Do they whine and cry when they have to 
learn new practices? 


- Are you at your wits end? 


You're Pregnant - Now What? 


- For better or worse, you've been given 
the responsibility to guide your 
organization to CMMI Level X 


- Parenting your organization will be the 
most rewarding (and most frustrating) 
thing you will even do 


- Starting talking to successful parents 
about what they did 


Si 


Sibling Rivalry - Introducing the 
New Baby 


- Baby CMMI will compete for favor among 
other initiatives in your organization 


- Talk to your significant other (the sponsor) 
about their expectations 
- Discipline, sharing duties, etc. 


- Plan a date night (about once a month) to 
review status against expectations 


The Terrible Two's 


- New babies are cute, but there are lots 
of diapers to change 


- As your CMML initiative matures, expect 
some pushback 
- "No!" can be a way of testing limits 


- Set firm groundrules 

- Destructive behavior (biting, hitting, refusing to 
share their artifacts and measures) must be 
stopped 


Establishing a Bed-Time Routine 


- Children need discipline to feel safe 
- Acceptable and unacceptable behavior 


- Establish a routine for working together 

- What is the improvement approach and timeline? 
- What is expected of the projects? By when? 

- How will progress be measured and reported? 

- What are the consequences? 


- A daily nap often improves the attitude 


Corporal Punishment: Pros and Cons 


- You can catch more flies with honey than 
you can with vinegar 


- Firm expectations and a regular routine 
will often eliminate the need for 
punishment 


- If you must spank, you and your sponsor 
should agree on the time and place 


Temper Tantrums 


- Losing their temper is your child's way of 
dealing with the loss of control 


- Remember to stay calm, and act like the 
adult 


- Do not tolerate physical and verbal abuse 
- Use time-out when needed 


Sharing with Others 


- With multiple siblings (projects), there is 
often a issue with sharing (artifacts, 
measures, progress, etc.) 


- Remind the projects that good families 
(organizations) share assets for the benefit 
of all 


- Giving up something now means getting something 
else when you need it 


- Maintain diligence to block access to 
dangerous items (e.g., non-compliant assets) 


Potty-Training Made Easy 


- Projects sometime grow dependent on your 
help for cleaning up the “dirty stuff” 


- Help projects understand that maturing is 
about taking responsibility for your own 
needs, and only asking when you need help 


- Reward them when you observe the 
desired behavior 


- Let them know “mistakes” are OK 


Si 


Separation Anxiety 


- As we approach the “big day” (appraisal), 
expect some clinging behaviors 

- What questions will they ask? 

- What should I say? 

- Can I just stay home? 


- Help them look forward to the experience 
- This is a chance to show what you can do 


- A dry-run may help relieve the anxiety 


Your Child May Never Become a 
Doctor 


- If you follow this advise, you child may 
grow up to be healthy and happy 


- Remember that all children are different, 
and each are special in their own way 


- Learn to love yours unconditionally, no 
matter how they turn out 


Si 


Contact Info 
Mrs. Doubtfire can be reached through: 


Rick Hefner 
Northrop Grumman Corporation 
rick. hefner@ngc.com 
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